"Training is a one-time expense." Now make that an engineering number.

Lesson 2 · reading the Cloudflare inference-vs-training article like a runtime engineer · your first cost model

You brought Cloudflare — AI inference vs. training. It draws the same line Lesson 0 drew, in the industry's standard words. This lesson takes its one load-bearing sentence and turns it into arithmetic you can run — because that sentence is the business case for the career you're building.

What the article gets right, in one line

"Training an AI model can be very expensive in terms of compute power. But it is more or less a one-time expense. … Inference, however, is ongoing." — Cloudflare, AI inference vs. training

That is the whole reason inference runtime engineering is a job. Training is a capital expense someone already paid. Inference is a bill that arrives every single day for as long as the product lives. A 10% kernel improvement on a one-time expense is a rounding error; a 10% improvement on a perpetual bill is a line item on someone's P&L. You are being hired against the recurring side.

Where the article stops, and your job starts

Cloudflare is a learning-centre explainer, not a runtime document. Read it for the vocabulary, then notice the three places it hands you a definition too coarse to engineer with:

The article says	True, but…	What you must model instead
Inference is "a trained model drawing conclusions from brand-new data" — a stop sign recognised, a prediction made.	That's the statistical definition. It describes one classification: input in, answer out.	For an LLM, one request is not one forward pass. It is one prefill pass over the prompt, then one forward pass per output token. A 256-token reply is 257 passes.
Inference "takes quite a bit of compute power and can be very expensive."	The bill is real, but "compute power" points at the wrong resource.	During decode, the GPU's math units sit near idle. The cost is memory traffic: re-reading every weight out of HBM for each token.
Training is "more or less a one-time expense."	Per-model, yes.	Whether serving passes that one-time bill is a tunable, not a fact. It depends on batch size and precision — the two knobs you're learning to turn.

None of this makes the article wrong. It makes it upstream of you. Your value starts exactly where its last paragraph ends.

The two phases hiding inside "an inference"

PREFILL — read the prompt

Runs once per request.
Processes all P prompt tokens together.
Big matmuls, weights read once for the whole prompt.
Lots of math per byte → usually compute-bound.
Sets your time to first token.

DECODE — write the reply

Runs once per output token.
Processes exactly one new token.
Same weights re-read from HBM every single step.
Almost no math per byte → memory-bound.
Sets your tokens per second.

Two numbers decide which side a phase lands on. Both are ratios you already think in — you just call them "CPU-bound vs I/O-bound".

The roofline test: compare the work's arithmetic intensity to the machine's balance point.

Listen first: two ratios. The first is arithmetic intensity — floating-point operations divided by bytes moved. It is a property of the work you are asking for. The second is machine balance — the GPU's peak operations per second divided by its memory bandwidth. It is a property of the hardware. On an A100 that is 312 trillion operations per second over 2039 gigabytes per second, which comes out to about 153 operations per byte. The rule: if the work's intensity is below the machine's balance, the kernel is memory-bound and the math units idle; if it is above, the kernel is compute-bound.

arithmetic intensity = FLOPs performed / bytes moved      # property of the WORK
machine balance      = peak FLOP/s / memory bandwidth     # property of the HARDWARE

A100 80GB:  312e12 FLOP/s  /  2039e9 B/s  ≈  153 FLOP/byte

intensity < 153  →  MEMORY-bound   (math units idle, waiting on HBM)
intensity > 153  →  COMPUTE-bound  (math units saturated)

Now the punchline. In decode, a batch of B sequences does 2 × N × B FLOPs while reading N × bytes_per_weight bytes, for an N-parameter model. The N cancels:

Decode arithmetic intensity, with the parameter count cancelled out.

Listen first: decode intensity equals two times the batch size, divided by the bytes per weight. The model size disappears entirely. In FP16, two bytes per weight, that is simply the batch size — so at batch one you get an intensity of one, against a machine that wants a hundred and fifty-three. You are running the GPU at well under one percent of its math capability, and the only fixes available are raising the batch size or shrinking the bytes per weight.

decode intensity = (2 × N × B) / (N × bytes_per_weight)
                 =  2 × B / bytes_per_weight        # N cancels — model size is irrelevant!

FP16 (2 bytes), batch 1  →  intensity 1     vs machine balance 153   ← 153x short
FP16 (2 bytes), batch 64 →  intensity 64    vs 153                   ← still short
INT4 (0.5 byte), batch 1 →  intensity 4     vs 153

Two consequences fall straight out of that one line, and they are the two levers the whole serving industry pulls: raise B (batching) and shrink bytes_per_weight (quantization — Netra Task A territory). Go feel them.

Micro-world: one request, all the way through

Roofline cost model — Llama-class model on one A100 80GB

Drive the knobs. Watch the internal view: what each phase is bound by, and how idle the math units are.

Model

Weight precision

Prompt tokens

Output tokens

Batch size: 1

Prefill Decode — the three panels and both tables below all show the same run

1 · Where each phase sits on the roofline

Both axes are log scale. The roof is the hardware's ceiling; the ridge at 153 FLOP/byte is where waiting on memory stops and doing arithmetic starts.

2 · Where the request's wall-clock actually goes

One full batch, start to finish. Prefill runs once; decode runs once per output token.

3 · One decode step: which resource is the ceiling

The bottleneck is whichever bar is pinned at 100%. The other one is what you are paying for and not using.

Table view

Per-phase roofline breakdown
Phase	FLOPs	Weight bytes read	Intensity	Bound by	Time	Math units busy
Prefill (×1)
Decode (×1 token)

Time to first token
Output speed, per user
Output speed, whole GPU
GPU-seconds billed per request
Requests until serving equals the entire training run
…at 1,000,000 requests/day, that is

Start at the defaults (7B, FP16, batch 1). In panel 3, the memory bar is pinned at 100% and the math bar is a stub at 0.65%. That picture is what kills the phrase "inference is expensive because it's a lot of math."
Drag the batch slider slowly and watch the orange decode dot in panel 1 climb the diagonal toward the ridge. That diagonal is the memory wall: while you're on it, every extra sequence in the batch is free arithmetic.
Stop at batch 64. Each decode step still takes the same 6.87 ms — but GPU-seconds per request collapses ~35×. You got cheaper without making the hardware faster.
Back to batch 1, switch to INT4. Identical FLOPs, ~4× faster decode. The dot jumps right along the roof without you adding a single operation. Proof the bottleneck was never arithmetic.
Hunt the ridge. Raise the batch until the decode dot reaches the corner and panel 3 flips — both bars at 100%. Note the number at FP16, then again at INT4. Quantization gets you there 4× sooner. Why?
Watch panel 2 invert. At batch 1, decode is 98.7% of the request. Push batch to 256 and prefill takes over — you've traded a memory problem for a compute problem, which is a completely different optimisation job.
Watch the last two table rows. At batch 1, serving burns Llama 2 7B's whole training bill in about a year. At batch 64 it takes almost four decades. Cloudflare's "one-time expense" is true or false depending on a slider you control.

What this model deliberately ignores: the KV cache (which is what actually caps batch size in production), attention FLOPs, kernel launch overhead, and the fact that real kernels never hit peak. It is a roofline estimate — the same back-of-envelope a serving engineer does before writing code, not a benchmark. Weight-only quantization is assumed: weights stored narrow, math still done in FP16 after dequantization, which is why the peak-FLOPs figure does not change with precision.

Check yourself

Cold-recall defense (one breath):
Training is capex, inference is opex — that's why the job exists. One LLM request is prefill (all P prompt tokens at once, compute-bound, sets time-to-first-token) plus one decode pass per output token (memory-bound, sets tokens/sec). Decode intensity is 2 × batch ÷ bytes_per_weight — the model size cancels — versus an A100 balance of ~153 FLOP/byte. At batch 1 in FP16 that's 1 vs 153: the math units are <1% busy and the GPU is just re-reading weights out of HBM. So the two levers are batching (raise B) and quantization (shrink the bytes).

Why this is the right lesson before the current problem

Every technique on your mission list is one of those two levers, or a way to buy back the memory traffic they cost:

Quantization / NF4 dequantization (Netra Task A) — shrink bytes_per_weight. You now know precisely why that speeds up decode and barely touches prefill.
Batching, continuous batching — raise B toward the balance point.
KV cache — the thing that stops you raising B, and the next mechanism worth a lesson.
Kernel fusion, and Triton itself — fewer round-trips to HBM per op. Memory-bound work is exactly where a fused kernel wins.

That is also why Lesson 1's vector add is the right first kernel: elementwise add is the purest memory-bound kernel there is. Its arithmetic intensity is about 1 FLOP per 12 bytes. When Experiment 0001 reports GB/s instead of TFLOP/s, this lesson is the reason.

Primary source

Read Cloudflare — AI inference vs. training (~5 min) for the vocabulary and the "ongoing vs one-time" framing. Trust it as an orientation piece (★★★☆☆), not as a runtime reference — everything below the token level in this lesson comes from the A100 datasheet (312 TFLOP/s FP16, 2039 GB/s) and the Llama 2 paper, Table 2 (7B: 184,320 A100-hours).

💬 I'm your teacher for this — ask me followups any time. The obvious next one: "what actually stops me setting batch to 153?" The answer is the KV cache, and it's the next lesson whenever you want it.