Resources

High-trust sources for grounding the AI inference lab. Triton is young and fast-moving, so prefer official docs, primary source material, reproducible benchmarks, and real codebases over generic roadmaps.

How to use these resources

• Use Netra-style tasks as the short-term benchmark.
• Use Triton/CUDA material for hands-on kernel skill.
• Use Baseten's inference material as the high-level map.
• Use real runtimes like sam3.c, llama.cpp, vLLM, and SGLang as architecture studies, but scope each study to one subsystem at a time.
• Avoid roadmap-driven wandering unless it directly supports an experiment.

Current target

• Netra Runtime — AI Engineer Interview Puzzles
• Practical benchmark for Triton kernels, quantization/dequantization, QLoRA/FSDP2, torch.compile, upstream PRs, and performance explanations.
• Use as a forcing function, not necessarily as the final employer target.

Primary / canonical

• Introducing Triton: Open-source GPU programming for neural networks — OpenAI (2021)
• Trust: ★★★★★ (the source of record). The phrase that grounds Lesson 0001: Triton uses an SPMD model "in which programs – rather than threads – are blocked." Kernels are "launched concurrently with different program_id's on a grid of so-called instances." ~10-min read.
• Triton documentation — Introduction — triton-lang.org
• Trust: ★★★★★ (official). Lists exactly what the compiler automates: "automatic coalescing, thread swizzling, pre-fetching, automatic vectorization, tensor core-aware instruction selection, shared memory allocation/synchronization, asynchronous copy scheduling."
• Triton tutorial — Vector Addition — triton-lang.org
• Trust: ★★★★★ (official). The canonical first kernel. Source of the annotated add_kernel in Lesson 0001.

Inference engineering map

• Inference Engineering — Baseten
• Use as a stack map: models, hardware, software, optimization techniques, modalities, and production. Do not publish standalone chapter summaries; connect reading to experiments.
• inferenceengineering.tech
• Interactive companion/course. Useful for reinforcement, but lower priority than implementing and benchmarking.

GPU fundamentals

• Programming Massively Parallel Processors
• Deep CUDA/GPU fundamentals: execution model, memory hierarchy, tiling, synchronization, performance reasoning, and parallel patterns.
• How it comes into play: use it as the why underneath Triton. Triton hides thread-level CUDA details, but Netra-style kernel work still requires the mental model: coalesced memory access, occupancy, cache behavior, shared memory, arithmetic intensity, and why a kernel is memory-bound or compute-bound.
• Read selectively after experiments raise questions. Example: run Triton vector add, then read about memory bandwidth/coalescing; attempt matmul, then read tiling/shared-memory chapters; work on dequantization, then read memory hierarchy and data movement. Avoid reading cover-to-cover before coding.

Runtime codebases to study

• NetraRuntime/sam3.c
• To study and personally verify. External-codebase summary says it is a pure-C SAM3 inference runtime with custom tensor ops, CPU/Metal backends, mmap weight loading, quantization support, feature caching, CLI/bindings, and benchmarks.
• Use it as a model-specific runtime engineering case study only after scoping one subsystem at a time, such as weight loading, feature caching, or a single backend kernel family.

Hardware & cost-model numbers (cite these, don't guess)

• NVIDIA A100 Tensor Core GPU datasheet
• Trust: ★★★★★ (vendor primary). A100 80GB SXM: 312 TFLOP/s FP16 tensor core (dense; 624 with sparsity), 2039 GB/s HBM2e. Those two numbers give the machine balance of ≈153 FLOP/byte used throughout Lesson 0002.
• Llama 2: Open Foundation and Fine-Tuned Chat Models — Meta (arXiv:2307.09288)
• Trust: ★★★★★ (primary). Table 2 is the citable training-capex number: 7B = 184,320 A100-80GB GPU-hours, 13B = 368,640, 70B = 1,720,320, total 3,311,616. Used as the denominator in the training-vs-serving crossover.

Background on the CUDA model (the diagram)

• Thread block (CUDA programming) — Wikipedia)
• Trust: ★★★☆☆ (good enough for the vocabulary in the diagram: threadIdx, blockIdx, blockDim, gridDim).

Secondary (orientation, read with care)

• AI inference vs. training: What is AI inference? — Cloudflare Learning Center
• Trust: ★★★☆☆. Correct, standard industry vocabulary for the training / inference split, and the source of the framing sentence Lesson 0002 is built on: training is "more or less a one-time expense", inference "however, is ongoing." Treat as orientation only — it defines inference statistically ("drawing conclusions from brand-new data"), never mentions prefill vs decode, and calls the serving bottleneck "compute power" when for LLM decode it is memory bandwidth. Blocks automated fetching (HTTP 403); read in a browser or via an archive snapshot.
• Triton Is Not CUDA in Python — It's a Tiling DSL — Medium
• Trust: ★★★☆☆. Useful framing ("one program processes a whole tile, not one value"), but verify any claim against the official docs above.

Communities (for wisdom — testing understanding with practitioners)

• GPU MODE (formerly CUDA MODE) Discord + YouTube lectures — active practitioner community for Triton/CUDA kernel work.
• Triton GitHub Discussions — questions answered by maintainers.
• r/CUDA and r/LocalLLaMA for inference-focused discussion.