Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →jpoindexter/inference-performance
Use when serving or optimizing LLM inference in production — diagnosing or improving TTFT/TPOT/throughput, choosing batching strategy, sizing GPUs, picking vLLM/TensorRT-LLM, or debugging low GPU utilization, TTFT spikes, and OOM. Covers prefill vs decode, the roofline, continuous batching, PagedAttention, chunked prefill, disaggregation, and FlashAttention.
Passed 0 of 21 test cases with the skill, and 21 of 21 without it.
1 fixed19 passed either way1 still failing1 broke
Tokens, turns and time describe resource use, not a dollar price.
The instructions your model reads when the skill loads.
Serving an autoregressive transformer is two workloads wearing one trench coat. Prefill processes the whole prompt at once; decode emits one token at a time. They have opposite bottlenecks, so a config tuned for one starves the other. Master this split and the rest (roofline, batching, paging) follows.
| Prefill | Decode | |
|---|---|---|
| Work | Process all P prompt tokens in parallel | Generate token t from tokens 0..t-1 |
| Matmul shape | [P, d] × [d, d] — tall, fat GEMM | [1, d] × [d, d] — GEMV per request |
| Bottleneck | Compute (FLOPS) | Memory bandwidth (HBM) |
| Passes per token | 1 (amortized over P tokens) | 1 full weight read per token |
| KV cache | Writes P tokens of K,V | Reads all prior K,V, appends 1 |
| Drives | TTFT (time to first token) | TPOT/ITL (time per output token) |
| Parallelism | Across the sequence (free) | Across the batch (must aggregate) |
| Scales with | Prompt length P (∝ FLOPs, ∝ P² attn) | Output length O (∝ steps) |
The asymmetry is the whole game. Prefill does ~P tokens of work in one weight-load, so it is compute-bound the moment P exceeds a few dozen. Decode loads all model weights to produce one token, so at batch 1 it is brutally memory-bound — the GPU's ALUs sit idle waiting on HBM. You fix them with different levers: prefill wants more FLOPS and work-splitting; decode wants more bandwidth and a bigger batch.
1/TPOT = per-stream tokens/s. The "reads smoothly" metric.TTFT + (O − 1) × TPOT for O output tokens. Long generations are TPOT-dominated; short ones are TTFT-dominated.tokens/s (decode) and req/s. The cost metric: $/token ∝ 1/throughput.Rule: TTFT is a prefill problem, TPOT is a decode problem, throughput is a batching problem. Never debug them with one knob.
The shape of a request — prompt length P and output length O — determines which metric dominates and which phase you pay for:
| Lever | TTFT | TPOT | E2E | Throughput | KV memory |
|---|---|---|---|---|---|
↑ prompt P | ↑↑ (∝ P, attn ∝ P²) | ~flat | ↑ | ↓ (prefill steals compute) | ↑ |
↑ output O | none | ~flat per token | ↑↑ (∝ O) | ↓ (long tail holds slot) | ↑ |
↑ batch B | ↑ (queueing) | ↑ (contention) | ↑ | ↑↑ | ↑ |
Two profiles need opposite tuning:
test cases passed, vs 21 / 21 without
More skills from the same source and nearby in the registry, with their headline benchmark lift.