---
name: jpoindexter/quantization-and-model-compression
source: https://app.decimal.ai/s/jpoindexter-quantization-and-model-compression@1/SKILL.md
source_sha256: 8aa71d876c9f
---

# Quantization & Model Compression

Compression is how a model that *trained* on a cluster *serves* on the GPU you can afford. Three orthogonal tools attack three different costs: **quantization** cuts memory/cost (maybe lossy), **speculative decoding** cuts decode latency (lossless), **distillation** produces a genuinely smaller model (training cost up front). Pick by the bottleneck, not by hype. They compose.

## 1. Why compress

- **Memory fit.** A 70B model in FP16 weights = ~140 GB — won't fit one 80 GB H100. At INT4 it's ~35 GB and fits with room for KV cache. Quantization is often the *only* way onto one GPU.
- **Cost.** Fewer/smaller GPUs per replica; higher batch density. Half the bytes per weight ≈ half the VRAM ≈ roughly half the $/token at fixed throughput.
- **Latency.** LLM decode is **memory-bandwidth bound**, not compute bound — every token streams all weights from HBM. Halving weight bytes ~halves the per-token read, so INT4/INT8 decode is faster even when the math is the same. Speculative decoding attacks the *same* bottleneck by verifying many tokens per forward pass.
- **Throughput / batch.** Smaller weights + smaller KV cache leave headroom for larger batches, which is where serving economics live.

> Rule: prefill is compute-bound (FLOPs), decode is bandwidth-bound (byte reads). Quantization helps decode most; it does little for prefill-heavy/long-prompt workloads.

## 2. Numeric formats

| Format | Bits | Range driver | Precision | Mem vs FP16 | Hardware | Notes |
|--------|------|--------------|-----------|-------------|----------|-------|
| **FP32** | 32 | 8 exp | 23 mantissa | 2× | universal | training/reference baseline |
| **FP16** | 16 | 5 exp | 10 mantissa | 1× | all modern GPU | narrow range — can overflow in training |
| **BF16** | 16 | 8 exp | 7 mantissa | 1× | A100+/TPU | FP32 range, less precision — **default training dtype** |
| **FP8 E4M3** | 8 | 4 exp | 3 mantissa | 0.5× | H100/Ada, MI300 | more precision, less range — **weights/activations**; ~lossless inference |
| **FP8 E5M2** | 8 | 5 exp | 2 mantissa | 0.5× | H100/Ada | more range, less precision — gradients/error-tolerant |
| **INT8** | 8 | scale/zero-point | uniform | 0.5× | universal (DP4A/tensor) | classic W8A8; needs outlier handling |
| **INT4** | 4 | scale/zero-point | uniform | 0.25× | via dequant kernels | weight-only sweet spot; quality risk on small models |

Key contrasts:
- **Float (FP8) vs integer (INT8):** floats put precision near zero where weights cluster, so FP8 tolerates outliers far better than INT8. On H100, FP8 inference is frequently **near-lossless** with almost no calibration — the easiest big win if you have the hardware.
- **BF16 vs FP16:** same bytes, but BF16 trades mantissa for exponent range. Train in BF16 (no loss scaling needed); FP16 still fine for inference.
- **INT4 isn't "half of INT8" in quality** — error roughly doubles per bit dropped and the cliff is nonlinear (see §6).
- **E4M3 vs E5M2:** E4M3 (range ±448) is the *forward* dtype (weights, activations) — more mantissa = better signal. E5M2 (range ~±57344) matches FP16's exponent and is for gradients/anything that can spike. Hopper tensor cores run both natively.

Hardware reality check:
- **FP8 tensor cores** exist only on Hopper (H100/H200), Ada (L4/L40S), and AMD MI300. Pre-Ampere has no FP8 — there, FP8 is software-emulated and *slower*, so don't.
- **INT8 (DP4A / INT8 tensor cores)** is near-universal (Turing onward) — the portable activation-quant target.
- **INT4** has no native matmul on most GPUs; kernels **dequantize INT4→FP16 on the fly** (e.g. Marlin, ExLlama). The win is memory-bandwidth (fewer bytes read), not raw INT4 math. A bad dequant kernel can erase the speedup — kernel choice matters as much as the format.

### Granularity, symmetry, what gets quantized
- **Weight-only (W4/W8A16):** quantize weights, keep activations FP16. Dominant for LLM *serving* — activations carry the nasty outliers, so leaving them in FP16 sidesteps the hardest problem. GPTQ, AWQ, bitsandbytes, GGUF are all weight-only.
- **Weight + activation (W8A8):** quantize both → INT8 matmuls, ~2× compute throughput, but you must tame activation outliers first (that's SmoothQuant's whole job).
- **Per-tensor** — one scale for the whole tensor. Fast, lossy; outliers dominate the scale.
- **Per-channel** — one scale per output channel. Standard for weights.
- **Per-group** — one scale per block of N weights (typically **group_size=128**). Best quality/size trade for INT4; the de-facto default.
- **Symmetric** (zero-point=0, range `[-a,a]`) — cheaper kernels, default for weights. **Asymmetric** (learned zero-point) — better for skewed/non-centered activations.

> Smaller group size = better accuracy, more scale metadata. group=128 ≈ +0.5–1% size for most of the quality. group=32 for fragile models; per-tensor only when kernels demand it.

### The integer-quant math (what a scale/zero-point actually is)
```
# asymmetric INT-n, real value r → quantized q
scale      = (r_max - r_min) / (2^n - 1)        # bucket width
zero_point = round(-r_min / scale)              # integer mapping of 0.0
q          = clamp(round(r / scale) + zero_point, 0, 2^n - 1)
r̂          = (q - zero_point) * scale           # dequantized (lossy)
```
Symmetric drops the zero-point (`r_min = -r_max`, zp=0) → cheaper kernels. The stored overhead per quantized block is just `{scale, zero_point}` — that's why group_size trades size for accuracy. INT4 weight-only "memory" = 4 bits/weight **plus** scales: at group=128 that's ~4.25 effective bits/weight, not a clean 4.

### Worked memory budget — Llama-3 70B
| Component | FP16 | INT8 W8A16 | INT4 W4A16 g128 |
|-----------|------|------------|-----------------|
| Weights (70B params) | ~140 GB | ~70 GB | ~37 GB |
| KV cache @ 8k ctx, bs=16 (FP16) | ~40 GB | ~40 GB | ~40 GB |
| **Total** | **fails on 1×80 GB** | fits, tight | **fits with room** |
| + FP8 KV cache | — | ~50 GB | ~57 GB |

Takeaway: INT4 weights are what *fit* a 70B on one H100; KV-cache quant is what lets you then grow batch/context. Quantizing weights alone won't save you if a long-context KV cache is the real wall.

## 3. PTQ vs QAT

| | **PTQ** (post-training) | **QAT** (quant-aware training) |
|---|---|---|
| Cost | minutes–hours, a few hundred calibration samples | full/partial retrain, days of GPU |
| Data | small calibration set | full training pipeline + labels |
| Quality at INT4 | good (AWQ/GPTQ) | best, recovers most of the cliff |
| When | **default — start here** | only after PTQ proves insufficient and accuracy is worth a retrain |

For LLMs, **PTQ is the norm** — QAT a 70B model is rarely worth it. QAT earns its cost for sub-4-bit, edge deployment, or when a specific eval must not regress.

## 4. The PTQ methods that matter

### GPTQ — second-order, layer-wise
Quantizes weights one layer at a time, using approximate **second-order (Hessian) information** from a calibration set to choose rounding that minimizes output error, compensating remaining columns as it goes. Strong **4-bit weight-only**; 3-bit possible with degradation.
- Use when: you want max accuracy at W4A16 and can spend calibration time (slower to produce than AWQ).
- Watch: overfits the calibration distribution — use representative data; one-shot, no backprop.

### AWQ — Activation-aware Weight Quantization
Insight: not all weights matter equally. The ~**0.1–1%** of weight channels that multiply large-magnitude activations are *salient* — quantizing them dumbly destroys quality. AWQ measures activation magnitude, then **per-channel scales** weights to protect the salient ones (no mixed precision needed). Excellent W4A16, robust on instruction-tuned models, fast calibration.
- Use when: 4-bit weight-only serving — often the **default** today (vLLM/TGI first-class, great quality/speed).
- Watch: still needs a calibration set matching the deployment domain.

### SmoothQuant — migrate outliers for W8A8
Activation outliers (a few channels 10–100× larger) break INT8 activation quantization. SmoothQuant **mathematically migrates** the difficulty from activations into weights via a per-channel smoothing factor `s` (offline, equivalence-preserving: `(X·diag(1/s))·(diag(s)·W)`), so *both* go INT8 cleanly. Enables true **W8A8** INT8 matmuls (compute speedup, not just memory).
- Use when: you want INT8 *activations* for throughput on INT8 tensor cores, not just weight-only memory savings.
- Watch: tune the migration strength `α` (≈0.5); it pairs with, not replaces, weight quantization.

### bitsandbytes — NF4 / QLoRA, 8-bit
- **LLM.int8()** — W8A16 with an outlier-aware mixed-precision matmul (outlier columns kept FP16). Zero-calibration drop-in load (`load_in_8bit`), but slower kernels than AWQ/GPTQ.
- **NF4 (4-bit NormalFloat)** — an information-theoretically optimal 4-bit dtype for normally-distributed weights; the backbone of **QLoRA**: load a frozen 4-bit base and train small FP16 LoRA adapters on top. The standard way to **fine-tune a 70B on a single 48 GB GPU**.
- Use when: fast prototyping, fine-tuning under VRAM limits. For raw serving throughput, AWQ/GPTQ kernels usually beat bnb.

### GGUF / llama.cpp — k-quants for CPU/edge/Mac
GGUF is the container; **k-quants** (`Q4_K_M`, `Q5_K_M`, `Q6_K`, `Q8_0`, `Q2_K`…) mix bit-widths across tensor types and use per-block scales. `_K_M` ≈ medium, the popular balance. Runs CPU + Apple Metal + partial GPU offload.
- Use when: local/Mac/CPU inference, Ollama, no datacenter GPU.
- Picking: **Q4_K_M** is the standard "good enough" default; Q5/Q6 if you have RAM and want headroom; **avoid Q2_K / Q3** except on huge models where there's redundancy to spare.

### Quick chooser
| Goal | Pick |
|------|------|
| H100/Ada, near-lossless, minimal effort | **FP8** |
| GPU serving, 4-bit, best quality | **AWQ** (or GPTQ) |
| INT8 activations for compute throughput | **SmoothQuant (W8A8)** |
| Fine-tune big model on small VRAM | **QLoRA / NF4 (bnb)** |
| Mac / CPU / local | **GGUF Q4_K_M** |

## 5. KV-cache quantization

The KV cache grows with batch × sequence length and at long context **dwarfs the weights** — it, not weights, becomes the memory and bandwidth wall. Quantize K and V tensors to **FP8 or INT8** (→ ~2× cache shrink, ~2× longer context or batch). FP8 KV is typically near-lossless and is the safe default on H100; INT8 KV usually needs per-token/per-channel scales. Cross-ref the KV-cache / attention skill for paged-attention and cache-sizing math.
- Watch: the **V** tensor and very long contexts are most sensitive; quantize K first, keep an eval on long-context retrieval.

## 6. When quantization HURTS quality

The failure is rarely uniform — it concentrates.

- **Activation outliers.** Emergent in models >~6.7B: a few feature dimensions with huge magnitudes. Per-tensor INT8 lets them dominate the scale and crush everything else → garbage. Fixes: keep activations FP16 (weight-only), or SmoothQuant, or mixed-precision (LLM.int8()).
- **INT4 on small models.** <~3B params have little redundancy to absorb error; INT4 can cost several points of accuracy. **The smaller the model, the more it bleeds** — prefer INT8 or FP8 for small models.
- **Long-context degradation.** Quantized KV + quantized weights compound over thousands of tokens; retrieval/needle tasks regress before short-prompt evals notice.
- **Reasoning & code are more sensitive** than chat. Multi-step math, long code generation, and tool-call argument formatting amplify small per-token errors into wrong final answers. Always include a coding + reasoning eval, not just perplexity.
- **Error accumulation.** Per-layer error compounds through depth; autoregression compounds it again across tokens. A 0.3% per-layer error is invisible on layer 1 and fatal by layer 60 on a 200-token generation.
- **The quality cliff.** Quality is flat from FP16→INT8→ often INT4, then **falls off a cliff** at 3-bit / 2-bit. Bits-per-weight is the lever; the cliff location depends on model size and method.

Rough quality vs bits (weight-only, good method like AWQ; deltas widen on small/reasoning models):
| Precision | Typical quality vs FP16 | When safe |
|-----------|------------------------|-----------|
| FP8 / INT8 | ~lossless (<0.5%) | almost always |
| INT4 g128 | ~0.5–2% drop | ≥7B, non-reasoning-critical |
| INT3 | several % | large models only |
| INT2 (Q2_K) | large / unstable | 70B+ where redundancy survives |

### Measuring — perplexity is weak
- **Perplexity is necessary, not sufficient.** It can move <1% while task accuracy craters — it averages over tokens and hides reasoning/format failures.
- **Use task evals** matching deployment: MMLU/ARC (knowledge), GSM8K/MATH (reasoning), HumanEval/MBPP (code), plus a domain set and long-context needle test.
- **A/B against the FP16 baseline** on *your* traffic. Watch tool-call JSON validity and instruction-following — quant breaks these before it breaks fluency.
- Set a regression budget *before* quantizing (e.g. "≤1% MMLU, 0 broken tool calls") and gate on it.

## 7. Speculative decoding — lossless latency

A small **draft** model proposes the next *k* tokens; the **target** model verifies all *k* in **one parallel forward pass**. Accepted tokens are kept; the first rejection resamples from the target. The output distribution is **provably identical to the target alone** — *lossless*, pure speedup. Typical **2–3× decode speedup** (more on easy/predictable text). It trades extra FLOPs (cheap, prefill-style parallel) for fewer sequential memory-bound steps.

| Variant | Drafter | Notes |
|---------|---------|-------|
| **Two-model** | separate small model (same tokenizer/family) | classic; needs a good aligned draft model |
| **Self-speculative** | the target's own early layers / skipped layers | no extra model to ship |
| **Medusa** | extra decoding "heads" on the target predict multiple future tokens | tree-attention verify; no separate model, light training |
| **EAGLE / EAGLE-2/3** | lightweight head predicting at the **feature** level + tree drafting | **highest acceptance** of the head-based methods; SOTA speedup |
| **n-gram / prompt lookup** | copies likely continuations from the prompt/context | zero model, zero training; great for **summarization/RAG/code-edit** where output echoes input |

- **Acceptance rate** is everything. Speedup ≈ accepted-tokens-per-verify. High when draft≈target (same family, in-distribution); low on creative/high-entropy text.
- **Draft sizing.** The draft must be **much cheaper** than the target — rule of thumb ~10–20× smaller (e.g. a 7B drafting for a 70B, or a 1B for a 13B). Too large a draft and its cost eats the savings even at high acceptance; too small and acceptance drops. The draft *must share the target's tokenizer* (or use a translation layer).
- **The arithmetic.** With acceptance `α` and `k` proposed tokens, expected accepted ≈ `(1-α^(k+1))/(1-α)`. At α=0.8, k=5 → ~3.4 tokens/verify ≈ ~3× fewer target passes. At α=0.4 the same k yields ~1.6 — barely above 1, and after draft overhead you can net *negative*.
- **Helps:** low-batch / latency-sensitive serving, predictable or input-echoing output (RAG, summarization, code edits), strong draft alignment.
- **Hurts:** **low acceptance makes it slower** — you pay draft cost + a wasted verify per reject. Also near-useless at **high batch sizes**, where the target is already compute-saturated and there's no idle bandwidth to reclaim. Greedy/low-temp decoding accepts more than high-temperature sampling.

## 8. Distillation — a different axis

Train a small **student** to mimic a large **teacher** — match teacher *soft logits/distributions* (richer signal than hard labels), optionally intermediate features. The result is a permanently smaller, faster, cheaper architecture (e.g. DistilBERT ~40% smaller/~60% faster at ~97% quality; modern LLM "distilled" variants train a small model on a frontier model's outputs).
What signal you match:
- **Response (logit) distillation** — KL-divergence between student and teacher output distributions, usually at **temperature T>1** to soften logits and expose the teacher's "dark knowledge" (relative probabilities of wrong answers). Loss ≈ `α·KL(soft) + (1-α)·CE(hard labels)`. The default for LLMs.
- **Feature distillation** — match intermediate hidden states / attention maps. More signal, but needs architectural alignment (or a projector). Used by DistilBERT-style work.
- **On-policy / sequence-level (modern LLM distillation)** — student generates, teacher scores/corrects its *own* trajectories. Avoids the exposure-bias mismatch of pure offline logit matching and is how today's small "distilled from a frontier model" chat models are built.

Properties:
- **Task-specific distillation** beats general: a small student matching the teacher on *your* domain often rivals the teacher there while being a fraction of the size.
- Costs real training compute and a teacher (or its outputs); risks **overfitting a narrow distribution** — strong on the distilled task, brittle off it.
- Distillation changes the *model*; quantization changes its *representation*. Orthogonal — you can distill **then** quantize the student.
- Watch the **license** on synthetic teacher outputs — many frontier-model ToS forbid training competing models on their generations.

## 9. Picking the tool — and combining

| Tool | Cuts | Lossless? | Up-front cost | Changes |
|------|------|-----------|---------------|---------|
| **Quantization** | memory, $/token, decode latency | usually slightly lossy | minutes (PTQ) | representation |
| **Speculative decoding** | decode latency only | **yes** | a draft model/heads | nothing (same output) |
| **Distillation** | size, latency, $ | no (new model) | training run | the model itself |

- **Memory-bound / can't fit?** → quantize (AWQ/GPTQ/FP8) + KV-cache quant.
- **Latency-bound, batch small, quality untouchable?** → speculative decoding.
- **Want a structurally smaller model for a known task?** → distill.
- **Combine:** distill → quantize the student → serve with speculative decoding (the quantized student can even *be* the draft model). Stack memory + latency + size wins.

## 10. Failure modes — quick reference

- **INT4 quality cliff** — small or reasoning-heavy model + 4-bit → multi-point drop. Mitigate: AWQ/GPTQ + group=128, or step up to INT8/FP8 for models <~7B.
- **2/3-bit collapse** — below INT4, output degrades fast except on very large redundant models. Reserve Q2_K for 70B+ only.
- **Activation-outlier blowup** — naive per-tensor W8A8 → gibberish. Use SmoothQuant or weight-only.
- **Calibration overfit** — GPTQ/AWQ tuned on mismatched data underperform in production. Calibrate on representative traffic.
- **Speculative decoding slower than baseline** — low acceptance (misaligned draft, high-entropy output) or high batch. Verify acceptance rate >~0.6 before shipping; disable at high batch.
- **Distillation overfits narrow** — student aces the distilled task, fails adjacent ones. Distill on a broad-enough distribution or scope deployment to match.
- **Perplexity-only validation** — ships a model that reads fine but fails tool calls / math. Always run task + format evals against the FP16 baseline.

## 11. Tooling & recipes

**Quantize for GPU serving (AWQ, 4-bit):** `autoawq` produces the checkpoint; serve directly in **vLLM** (`--quantization awq`) or **TGI**. Calibrate on ~128–512 samples of representative traffic; group_size=128.
```python
# AWQ quantize, then load — illustrative
from awq import AutoAWQForCausalLM
m = AutoAWQForCausalLM.from_pretrained("meta-llama/Llama-3-70B-Instruct")
m.quantize(tokenizer, quant_config={"w_bit": 4, "q_group_size": 128, "version": "GEMM"})
m.save_quantized("llama3-70b-awq")
# serve: vllm serve llama3-70b-awq --quantization awq --kv-cache-dtype fp8
```
**FP8 (H100/Ada):** simplest path — vLLM `--quantization fp8` or NVIDIA **TensorRT-LLM** / Modelopt; near-lossless, minimal calibration.
**QLoRA fine-tune:** `transformers` + `bitsandbytes` `BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=bf16)` + `peft` LoRA adapters.
**Local/Mac:** `llama.cpp` `convert_hf_to_gguf.py` → `quantize model.gguf model-Q4_K_M.gguf Q4_K_M`, or pull a pre-quantized GGUF in **Ollama / LM Studio**.
**Speculative decoding:** vLLM `--speculative-model <draft>` (two-model), or `--speculative-model [ngram]` for prompt-lookup, or ship EAGLE/Medusa heads where the engine supports them.

**Decision flow:**
1. On H100/Ada? → try **FP8** first (lowest effort, near-lossless). Done if it passes evals.
2. Need to fit / cut cost further? → **AWQ/GPTQ W4A16 g128** + **FP8 KV cache**.
3. Want INT8 *activation* throughput? → **SmoothQuant W8A8**.
4. Latency-bound at low batch, quality fixed? → add **speculative decoding** (check acceptance >0.6).
5. Fine-tuning under VRAM limits? → **QLoRA/NF4**.
6. Need a structurally smaller model for a known task? → **distill**, then quantize the student.
7. Gate every step on a **task-eval regression budget** vs the FP16 baseline.

## Do / Don't

**Do**
- Start at FP8 (if H100/Ada) or INT8 — only drop to INT4 when memory forces it.
- Use AWQ or GPTQ with **group_size=128** for 4-bit weight-only serving.
- Quantize the KV cache (FP8) before quantizing weights harder, at long context.
- Validate with **task + format evals** and a regression budget set beforehand.
- Combine tools: distill → quantize → speculative-decode for stacked wins.

**Don't**
- Don't INT4 a <3B model and expect FP16 quality.
- Don't quantize activations per-tensor without taming outliers (SmoothQuant first).
- Don't trust perplexity alone, or evals on data unlike production.
- Don't ship speculative decoding without checking the acceptance rate, or at high batch.
- Don't reach for QAT before PTQ has demonstrably failed.