The inference cost model, on one page

Reference · roofline arithmetic for LLM serving · the back-of-envelope you do before writing a kernel

The one-line model

Prefill runs once over all P prompt tokens → compute-bound → sets time-to-first-token. Decode runs once per output token, re-reading every weight from HBM → memory-bound → sets tokens/sec. Decode intensity is 2 × B ÷ bytes_per_weight; the model size cancels.

Glossary (adhere to this in every lesson)

Term	Meaning
Prefill	The single forward pass over the whole prompt at the start of a request. Large matmuls; weights read once for all P tokens.
Decode	One forward pass per generated token. Each pass re-reads the full weight set for one new token per sequence.
TTFT	Time to first token — essentially the prefill time. The latency the user feels before anything appears.
Arithmetic intensity	FLOPs performed ÷ bytes moved. A property of the work. Units: FLOP/byte.
Machine balance	Peak FLOP/s ÷ memory bandwidth. A property of the hardware. A100: ≈153 FLOP/byte.
Memory-bound	Intensity below machine balance. Time = bytes ÷ bandwidth. Math units idle. Fix by moving fewer bytes.
Compute-bound	Intensity above machine balance. Time = FLOPs ÷ peak. Fix by doing less arithmetic (or using faster units).
Roofline	The model that says runtime = max(compute time, memory time). Deliberately optimistic; a floor on time, not a benchmark.
HBM	High-bandwidth memory — the GPU's off-chip DRAM. Fast in absolute terms, still the decode bottleneck.
Weight-only quantization	Store weights narrow (INT8/INT4/NF4), dequantize to FP16 for the math. Cuts bytes, not FLOPs — which is why it helps decode and barely helps prefill.
KV cache	Per-sequence cached attention keys/values. Grows with batch × context, and is what caps B in practice.

The formulas

The full roofline chain for an N-parameter model, batch B, prompt P, output O.

Listen first: six steps. One, weight bytes equals the parameter count times bytes per weight. Two, prefill floating-point operations equal two times parameters times prompt length times batch. Three, decode operations per step equal two times parameters times batch. Four, each phase's time is the larger of its operations divided by peak throughput, and its bytes divided by bandwidth — whichever resource is slower wins. Five, the whole batch takes prefill time plus output length times decode step time, and per-request cost is that divided by the batch size. Six, the crossover point is total training seconds divided by per-request seconds — how many requests it takes for serving to cost what training cost.

W_bytes   = N × bytes_per_weight              # weights streamed once per pass

F_prefill = 2 × N × P × B                     # whole prompt, whole batch
F_decode  = 2 × N × B                         # ONE new token per sequence

t_phase   = max( F / PEAK_FLOPS ,  W_bytes / BANDWIDTH )   # roofline

t_batch   = t_prefill + O × t_decode
t_request = t_batch / B                       # GPU-seconds billed per request

crossover = training_GPU_seconds / t_request  # requests until serving = training

The numbers worth memorizing (A100 80GB SXM)

Quantity	Value	Why it matters
Peak FP16 tensor-core throughput	312 TFLOP/s	Numerator of machine balance.
HBM2e bandwidth	2039 GB/s	Denominator. The decode bottleneck.
Machine balance	≈153 FLOP/byte	The line every kernel is judged against.
Batch to make FP16 decode compute-bound	≈154	`2B/2 > 153`. Below it you are wasting silicon.
Batch to make INT4 decode compute-bound	≈39	`2B/0.5 > 153`. Quantization gets you there 4× sooner.
Llama 2 7B decode, FP16, batch 1	6.9 ms/token · 0.65% math busy	The canonical "the GPU is idle and still slow" data point.
Llama 2 7B pretraining	184,320 A100-hours	The capex number serving is measured against.

NVIDIA A100 datasheet↗ Llama 2, Table 2↗

The decision table

Symptom	What it means	Lever
Low tokens/sec, math units near idle	Decode is memory-bound (the normal case)	Raise batch size; quantize weights; fuse kernels to cut HBM round-trips.
Slow time-to-first-token	Prefill is compute-bound and the prompt is long	Chunked prefill, prompt caching, better matmul kernels.
Can't raise batch size	KV cache has eaten the HBM	Paged/quantized KV cache, shorter contexts, bigger GPU.
Quantized and prefill barely improved	Expected — quantization cuts bytes, not FLOPs	Nothing wrong. Measure decode, not the whole request.

Health warnings

The roofline is a lower bound on time. Real kernels miss peak; a 60–70% match is a good kernel.
Attention FLOPs and KV-cache traffic are excluded here — they grow with context length and eventually stop being ignorable.
Peak FLOP/s is quoted dense. Sparsity figures (624 TFLOP/s) apply only to structured-sparse models.
A crossover number is a compute-time comparison, not a dollar comparison — training and serving hardware are rented differently.

💬 Ask your teacher to expand any row of these tables into its own lesson. The KV cache row is the one that is overdue.

Where this comes from

Lesson 2 — Inference is the half you pay for forever (has the interactive version of this page)
Reference — CUDA ↔ Triton cheat-sheet

Reference · Zain's AI Inference Lab · sources: NVIDIA A100 datasheet, Llama 2 (arXiv:2307.09288)