Reference-grade guide to shrinking and speeding up LLMs without retraining from scratch — numeric formats (FP8/INT8/INT4), PTQ methods (GPTQ, AWQ, SmoothQuant, bitsandbytes NF4, GGUF k-quants), KV-cache quantization, speculative decoding (Medusa/EAGLE/n-gram), and distillation — with concrete numbers, when each fits, and the quality cliffs.
SkillScore91Maintained
Automatic
◆AI Engineer pack
Last verified 8/3/2026
What it changes
Passed 20 of 22 test cases with the skill, and 18 of 22 without it.
Without skill82%
With skill91%
Checks it passed that the model alone missed
Recommends combining NF4 base loading with low-rank adapters (QLoRA) via bitsandbytes.
Requires testing structured tool-call JSON format validity against the unquantized FP16 baseline.
2 fixed18 passed either way2 still failing0 broke
Pass-rate lift
+9 pp
Tokens
+196%
Agent turns
0%
Response time
-15%
Model gemini-3.6-flashMeasured 8/3/2026v120 of 22 cases passed with skill
Tokens, turns and time describe resource use, not a dollar price.
What's inside
The instructions your model reads when the skill loads.
SKILL.md238 linessingle file
Quantization & Model Compression
Compression is how a model that trained on a cluster serves on the GPU you can afford. Three orthogonal tools attack three different costs: quantization cuts memory/cost (maybe lossy), speculative decoding cuts decode latency (lossless), distillation produces a genuinely smaller model (training cost up front). Pick by the bottleneck, not by hype. They compose.
1. Why compress
Memory fit. A 70B model in FP16 weights = ~140 GB — won't fit one 80 GB H100. At INT4 it's ~35 GB and fits with room for KV cache. Quantization is often the only way onto one GPU.
Cost. Fewer/smaller GPUs per replica; higher batch density. Half the bytes per weight ≈ half the VRAM ≈ roughly half the $/token at fixed throughput.
Latency. LLM decode is memory-bandwidth bound, not compute bound — every token streams all weights from HBM. Halving weight bytes ~halves the per-token read, so INT4/INT8 decode is faster even when the math is the same. Speculative decoding attacks the same bottleneck by verifying many tokens per forward pass.
Throughput / batch. Smaller weights + smaller KV cache leave headroom for larger batches, which is where serving economics live.
Rule: prefill is compute-bound (FLOPs), decode is bandwidth-bound (byte reads). Quantization helps decode most; it does little for prefill-heavy/long-prompt workloads.
2. Numeric formats
Format
Bits
Range driver
Precision
Mem vs FP16
Hardware
Notes
FP32
32
8 exp
23 mantissa
2×
universal
training/reference baseline
FP16
16
5 exp
10 mantissa
1×
all modern GPU
narrow range — can overflow in training
BF16
16
8 exp
7 mantissa
1×
A100+/TPU
FP32 range, less precision — default training dtype
FP8 E4M3
8
4 exp
3 mantissa
0.5×
H100/Ada, MI300
more precision, less range — weights/activations; ~lossless inference
FP8 E5M2
8
5 exp
2 mantissa
0.5×
H100/Ada
more range, less precision — gradients/error-tolerant
INT8
8
scale/zero-point
uniform
0.5×
universal (DP4A/tensor)
classic W8A8; needs outlier handling
INT4
4
scale/zero-point
uniform
0.25×
via dequant kernels
weight-only sweet spot; quality risk on small models
Key contrasts:
Float (FP8) vs integer (INT8): floats put precision near zero where weights cluster, so FP8 tolerates outliers far better than INT8. On H100, FP8 inference is frequently near-lossless with almost no calibration — the easiest big win if you have the hardware.
BF16 vs FP16: same bytes, but BF16 trades mantissa for exponent range. Train in BF16 (no loss scaling needed); FP16 still fine for inference.
INT4 isn't "half of INT8" in quality — error roughly doubles per bit dropped and the cliff is nonlinear (see §6).
E4M3 vs E5M2: E4M3 (range ±448) is the forward dtype (weights, activations) — more mantissa = better signal. E5M2 (range ~±57344) matches FP16's exponent and is for gradients/anything that can spike. Hopper tensor cores run both natively.
Hardware reality check:
FP8 tensor cores exist only on Hopper (H100/H200), Ada (L4/L40S), and AMD MI300. Pre-Ampere has no FP8 — there, FP8 is software-emulated and slower, so don't.
INT8 (DP4A / INT8 tensor cores) is near-universal (Turing onward) — the portable activation-quant target.
INT4 has no native matmul on most GPUs; kernels dequantize INT4→FP16 on the fly (e.g. Marlin, ExLlama). The win is memory-bandwidth (fewer bytes read), not raw INT4 math. A bad dequant kernel can erase the speedup — kernel choice matters as much as the format.
Granularity, symmetry, what gets quantized
Weight-only (W4/W8A16): quantize weights, keep activations FP16. Dominant for LLM serving — activations carry the nasty outliers, so leaving them in FP16 sidesteps the hardest problem. GPTQ, AWQ, bitsandbytes, GGUF are all weight-only.
Weight + activation (W8A8): quantize both → INT8 matmuls, ~2× compute throughput, but you must tame activation outliers first (that's SmoothQuant's whole job).
Per-tensor — one scale for the whole tensor. Fast, lossy; outliers dominate the scale.