--- name: jpoindexter/adaptation-strategies source: https://app.decimal.ai/s/jpoindexter-adaptation-strategies@1/SKILL.md source_sha256: 6beb613c8b79 --- # Adaptation Strategies — Fine-tune vs ICL vs RAG vs Distillation You have a base model. It doesn't do the task well enough. You have four levers to change that, and they are **not interchangeable** — each changes a different thing. Picking the wrong one is the most expensive recoverable mistake in applied AI: weeks of labeling, a training run, and an eval suite, all to solve a problem a 200-token prompt would have fixed. The whole discipline reduces to one question: **what kind of gap do you have?** | Gap | Symptom | Right lever | |-----|---------|-------------| | **Knowledge** | Model doesn't *know* a fact, doc, or current state | **RAG** | | **Behavior / format** | Model knows enough but won't reliably *do it the way you need* (tone, structure, JSON, domain register) | **Fine-tune** (after prompt+RAG exhausted) | | **Capability** | Model literally *can't* — reasoning depth, hard task it fails at any prompt | **Bigger model**, then **ICL** to steer it | | **Cost / latency at scale** | Quality is fine, the bill or the p95 isn't | **Distill** or **fine-tune a small model** | Memorize that table. Everything below is detail on each lever and the failure modes of using one in the wrong row. --- ## 1. The four levers — what each actually changes ### In-context learning (ICL) Prompt engineering, few-shot examples, instructions, in-prompt context. **Changes behavior at inference time only — zero weight updates.** - **Mechanism:** the model conditions on tokens you supply. No training, no artifact, nothing persisted. - **Iteration speed:** instant. Edit a string, rerun. This is its superpower — you can try 50 variants in an hour. - **Cost shape:** pay in tokens on *every single call*, forever. A 2,000-token few-shot block on 10M calls/month is a real line item. - **Ceiling:** bounded by context window and by the fact that long prompts dilute attention and raise latency. Few-shot helps format and style enormously; it does *not* reliably add deep capability the base model lacks. - **Reach for it:** always first. Every other lever should be benchmarked *against a seriously-tried prompt*, not a lazy one. ### RAG (retrieval-augmented generation) Retrieve relevant chunks from a knowledge source and inject them into the prompt. **For KNOWLEDGE, freshness, and attribution — not for behavior.** (Full retrieval/chunking/eval detail lives in the **`rag` skill**; here we only place it among the levers.) - **Mechanism:** index → retrieve → stuff into context → generate. Weights untouched. - **What it fixes:** "the model doesn't know our docs / today's prices / this customer's history," and "I need a citation for every claim." - **What it does NOT fix:** how the model writes, formats, or reasons. Retrieving better text does not change tone or output structure. - **Freshness:** update the index, not the model. This is why current/changing facts belong in RAG, never in weights. ### Fine-tuning Update the model's weights on task data. **For BEHAVIOR, format, style, domain register, and latency-by-shrinking-the-prompt.** Variants: | Variant | What it does | When | |---------|--------------|------| | **Full SFT** | Updates all weights | Rare now; large data + budget + ops, max control | | **LoRA** | Trains small low-rank adapter matrices, freezes base | Default for behavior/format adaptation | | **QLoRA** | LoRA on a 4-bit-quantized base | LoRA when VRAM-constrained (fine-tune a 70B on one big GPU) | | **DPO / preference tuning** | Aligns to *preferred vs rejected* pairs, no reward model | Shape *judgment* — "prefer concise," "refuse X" — not just imitate | - **Mechanism:** gradient descent bakes the pattern into weights. The behavior becomes free at inference (no prompt tokens spent on it). - **What it's *great* at:** consistent output format, house style/tone, terse domain register, classification, function-calling shape, **and shrinking a giant prompt into the weights** so each call is cheaper/faster. - **What it's *bad* at:** reliably injecting facts (see §3), and anything you haven't built an eval for. - **Cost shape:** pay once to train (+ retrains on drift), cheap per call. The inverse of ICL. ### Capability gap — the lever nobody wants to hear A **capability gap** is when the base model *can't do the task at any prompt* — the reasoning is too deep, the chain too long, the judgment too hard. This is the one row where adaptation can't save you. - **Fine-tuning does not add capability.** It moves *format/behavior*, not the reasoning ceiling. You can fine-tune a model to *look* confident on a task it fundamentally can't do — that's how you get fluent, well-formatted wrong answers. - **RAG does not add capability.** It moves *knowledge into context*; it doesn't make the model reason better over that context. - **The only real fixes:** a **bigger/stronger base model**, or **ICL that decomposes the task** — chain-of-thought, step-by-step, breaking one hard call into several easy ones. CoT/decomposition is *steering* the capability the model already has into reach; it is not *adding* capability. If decomposition + the strongest model still fails, the task is out of reach and no adaptation lever changes that. ### Distillation Train a **small** model on a **big** model's outputs for a **specific** task. **For cost/latency at a fixed quality bar on a narrow task.** - **Mechanism:** generate a dataset with the teacher (or use its logits/rationales), fine-tune the student on it. The student learns *the task*, not the teacher's whole generality. - **What it buys:** 10–50× cheaper/faster inference at near-teacher quality — *on that one task*. The student does not inherit the teacher's broad ability. - **Precondition:** a **stable, well-defined task** and an eval that proves the student matches the teacher where it counts. No stable task → nothing to distill *to*. --- ## 2. The decision — gap → lever ``` ┌─────────────────────────────────────┐ │ Start: model underperforms on task │ └──────────────────┬──────────────────┘ ▼ ┌───────────────────────────────────────────────┐ │ Have you SERIOUSLY tried prompt + few-shot? │ └───────────────┬───────────────────────┬───────┘ no │ yes │ ▼ ▼ Do that first. Cheapest ┌──────────────────────────┐ fix, instant iteration. │ What kind of gap remains? │ └─────────────┬────────────┘ ┌──────────────────┬─────────────────────┬─────┴───────────────┐ ▼ ▼ ▼ ▼ KNOWLEDGE BEHAVIOR/FORMAT CAPABILITY COST/LATENCY (missing facts, (won't follow style, (can't do the task (quality OK, bill freshness, cites) format, tone) at any prompt) or p95 too high) │ │ │ │ ▼ ▼ ▼ ▼ RAG Fine-tune (LoRA/DPO) Bigger/better model Distill teacher→ (update index, on 100s–1000s of OR more ICL/CoT. student, OR fine- not weights) curated examples. Fine-tuning will tune a small model Need an eval set. NOT add capability on the stable task. the base lacks. ``` **Read it as a priority order, not a menu.** Cheapest, most reversible levers first: **prompt → RAG → fine-tune small → distill → full fine-tune**. Climb only when the cheaper rung is genuinely exhausted *against a real eval*, not against a hunch. A blunter heuristic: - **knowledge gap → RAG** - **behavior/format gap → fine-tune** - **capability gap → bigger model (then ICL to steer it)** - **cost/latency at scale → distill, or fine-tune a small model** --- ## 3. When each is the WRONG tool (this is where money dies) | You're tempted to… | Don't, because… | Do instead | |--------------------|-----------------|------------| | **Fine-tune to add facts** | Fine-tuning does *not* reliably inject knowledge — the model parrots the *form* of your examples and still hallucinates the content. Worse, it *forgets* other things (§5). Facts also go stale the day after you train. | **RAG.** Knowledge belongs in a retrievable index, not in weights. | | **RAG to fix tone/format** | Retrieving better passages changes *what* the model sees, not *how* it writes. You'll get correct facts in the wrong shape. | **Prompt** (cheap) or **fine-tune** (if the prompt fix doesn't stick at scale). | | **Fine-tune before exhausting prompt + RAG** | Premature: expensive, slow to iterate, and brittle — you've frozen a guess into weights and now own retrains forever. Most "we need to fine-tune" problems are a weak prompt or missing context. | Exhaust **prompt → RAG** first. Fine-tune only when a *good* prompt provably can't get there or is too expensive per call. | | **Distill without a stable task** | If the task definition still moves, the student is obsolete on arrival and you can't even tell — the eval keeps changing. | Stabilize the task + eval *first*. Distill last, when quality is settled and only cost/latency remains. | | **Keep using ICL when the prompt is huge & repeated** | A 3K-token instruction block sent on every call is a permanent tax on cost *and* latency, and long prompts dilute attention. | **Fine-tune** to bake the stable instructions into weights; calls get shorter, cheaper, faster. | | **Reach for a bigger model for a format problem** | Capability isn't the gap; you'll overpay for reasoning you don't need and still get inconsistent format. | **Fine-tune** a *small* model on the format. | The single most common and most expensive error: **fine-tuning to make the model "know" something.** It is the wrong tool by mechanism, not by degree. Knowledge → RAG. Always. --- ## 4. Fine-tuning mechanics you actually need ### PEFT / LoRA / QLoRA - **PEFT** (parameter-efficient fine-tuning) = train a tiny fraction of params, freeze the rest. LoRA is the dominant member. - **LoRA:** inserts low-rank matrices (rank `r`, typ. 8–64) into attention/MLP layers; only those train. ~0.1–1% of params updated. Cheap, fast, and you can **swap adapters** per task without storing full model copies. - **QLoRA:** base weights quantized to 4-bit (NF4) + LoRA adapters in higher precision. Lets you fine-tune a 70B model on a single 48–80GB GPU. Near-LoRA quality at a fraction of the memory. - **Default choice:** LoRA (or QLoRA if VRAM-bound). Full SFT only when you have lots of data, budget, ops maturity, and a measured reason adapters aren't enough. ### Preference tuning (DPO and kin) - **SFT teaches imitation** ("produce outputs like these"). **DPO teaches judgment** ("prefer this over that") from `(prompt, chosen, rejected)` triples. No separate reward model, simpler than RLHF/PPO. - Use DPO/ORPO/KTO when "good" is about *ranking* — conciseness, safety refusals, helpfulness — that's hard to specify by example alone. Common pattern: **SFT first to set the format, then DPO to refine the judgment.** ### Data: quality ≫ quantity - **Quality dominates.** A few hundred *clean, consistent, on-distribution* examples beat tens of thousands of noisy ones. The model imitates your data faithfully — including its mistakes, its inconsistencies, and its format drift. - **Rough scale:** LoRA behavior/format/classification often lands in the **low hundreds to low thousands** of examples. More data with worse consistency is a downgrade, not an upgrade. - **Consistency is the asset:** every example formatted the same way, labeled by the same rubric, reflecting the *exact* behavior you want at inference. One inconsistent labeler poisons the run. - **Match the inference distribution:** train on inputs that look like production. Train on clean prose, deploy on messy user text → it breaks. ### Eval-driven, always - **No held-out eval set = do not fine-tune.** Without it you cannot tell improvement from regression, and you *will* regress something. Build the eval before the training data. - **Hard split:** train / eval examples must never overlap. Leakage (§5) makes a broken model look perfect. - Eval the **whole behavior**, not just the target task — including a regression check on general capability to catch forgetting. --- ## 5. Failure modes — name them, build guards against them | Failure | What it looks like | Guard | |---------|--------------------|-------| | **Fine-tuning to memorize facts** | Confident, fluent, *wrong* answers; right format, fabricated content | Put knowledge in **RAG**; fine-tune only for behavior | | **No eval set** | "Seems better" with no number; can't defend the change | Build held-out eval **before** training; gate the run on it | | **Data leakage (train ↔ eval)** | Eval score great, production score bad | Dedup + split by source/entity before training; verify zero overlap | | **Overfitting** | Memorizes train examples, fails near-duplicates; brittle to phrasing | Fewer epochs, lower LoRA rank, more diverse data, early-stop on eval loss | | **Catastrophic forgetting** | Now great at the task, *worse* at things it used to do | PEFT (less invasive than full SFT), lower LR, mix in general data, regression-test broad ability | | **Drift** | Quality silently decays as the world / inputs move | Monitor production metrics; schedule retrains; keep volatile facts in RAG, not weights | | **Distilling an unstable task** | Student ships already-obsolete; can't tell because eval moves | Freeze task + eval first; distill last | | **Retrain treadmill underestimated** | "One training run" becomes a quarterly ops burden | Budget the *lifecycle* (data refresh, retrain, re-eval, redeploy), not just run #1 | **Catastrophic forgetting** deserves its own emphasis: weights are shared, so teaching one thing can erase another. It is the hidden cost that makes fine-tuning-for-facts doubly wrong — you pay to *not* learn the fact *and* to forget something useful. Mitigate with PEFT, conservative learning rates, mixing in general-purpose data, and a broad regression eval after every run. --- ## 6. Combining levers (the real-world answer is usually "and," not "or") These are not exclusive. Production systems stack them: - **RAG + fine-tuned small model** — the most common mature pattern. RAG supplies fresh, attributable *knowledge*; the small fine-tune nails *format/tone* and makes each call cheap. Knowledge stays swappable in the index; behavior stays baked in the weights. Best of both, and neither does the other's job. - **ICL + RAG** — retrieved chunks plus a few-shot template and instructions. The default starting architecture before any training exists. - **SFT → DPO** — imitate the format, then refine the judgment. - **Distill (teacher→student) + RAG** — student handles the narrow task cheaply; RAG keeps it factually current without retraining. - **Fine-tune to shrink the prompt, then keep ICL for the variable part** — bake the stable 3K-token instruction block into weights; keep a small dynamic few-shot slot for per-request specifics. Guiding split when combining: **knowledge → retrieval (mutable, external); behavior → weights (stable, internal).** Put each thing where it belongs and they stop fighting. --- ## 7. Worked example — adjudicating a real decision **Scenario:** a customer-support assistant. It must (a) answer using the *current* product docs and *this* customer's plan/usage, and (b) reply in the company's terse, no-fluff house voice with a fixed structure (answer → next step → doc link). The lazy instinct is "fine-tune it on our support transcripts." Walk the gaps instead: 1. **Knowledge gap?** Yes — current docs, per-customer state. That is a *knowledge* row → **RAG.** Index the docs + pull the customer record at query time. 2. **The trap, named:** fine-tuning on transcripts to "teach the product" is the §3 error in full. The facts go **stale** the day docs change; the model will **hallucinate** plausible-but-wrong details in perfect house format; and the run will **forget** unrelated ability. You'd pay to make the model *worse and more confident*. 3. **Behavior gap?** Yes — house voice + fixed structure won't hold from prompting alone across millions of varied tickets. That is a *behavior* row → **LoRA fine-tune** on a few hundred clean, consistently-formatted exemplar replies. This also **shrinks the prompt**: the style instructions move into weights, so every call is cheaper and faster. 4. **Capability gap?** No. A strong base model can do support reasoning given the right context. No bigger model needed. **Resolution: RAG + LoRA together** (§6's canonical pair). Knowledge lives in the swappable index and stays fresh; behavior lives in the adapter and stays consistent. Each lever does its own job and they stop fighting. **Mini-case (cost only):** a field-extraction step where a good prompt already hits the quality bar — but it's 10M calls/month on a frontier model. Quality gap = zero; the entire problem is cost/latency. → **Distill** the frontier model's outputs into a small student (or fine-tune a small model directly) once the schema is frozen. No new quality is needed, so no bigger model and no RAG — just move the *same* quality onto a cheaper engine. --- ## 8. Comparison table — the one-screen summary | Dimension | In-context learning | RAG | Fine-tuning (LoRA/QLoRA) | Distillation | |-----------|---------------------|-----|--------------------------|--------------| | **Changes** | Behavior at inference | Knowledge in context | Weights (behavior baked in) | A small model's weights | | **Solves the gap** | Capability-steering, quick format | Knowledge, freshness, citation | Behavior, format, style, tone, prompt-shrink | Cost/latency at fixed quality | | **Training cost** | None | None (build index once) | Moderate (one run, repeated on drift) | Moderate–high (dataset + run) | | **Per-call cost** | High (tokens every call) | Medium (retrieval + context tokens) | Low (short prompt) | Lowest (small model) | | **Latency** | Higher (long prompts) | Medium (retrieval hop) | Low | Lowest | | **Iteration speed** | Instant | Fast (re-index) | Slow (label → train → eval) | Slowest (needs stable task) | | **Freshness** | Per-call | Excellent (update index) | Poor (stale until retrain) | Poor (retrain) | | **Data needed** | A few examples | A corpus to index | 100s–1000s clean examples + eval | Teacher outputs + eval | | **Reversibility** | Trivial (edit string) | Easy (swap index) | Hard (own retrains, forgetting) | Hard (rebuild dataset) | | **Ops burden** | None | Index pipeline, retrieval eval | Train + eval + monitor + retrain | Same as fine-tune + teacher | --- ## 9. Operating rules (do / don't) **Do** - Try a *serious* prompt + few-shot first, and benchmark every other lever against it on the same eval. - Build the held-out eval set **before** you collect training data or stand up retrieval. - Put facts/knowledge in **RAG**; put behavior/format/tone in **weights**. - Prefer **LoRA/QLoRA** over full SFT; prefer fine-tuning a **small** model over reaching for a bigger one when the gap is format, not capability. - Treat **data quality and consistency** as the primary lever — a few hundred clean examples beat thousands of noisy ones. - Regression-test broad capability after every fine-tune to catch forgetting. - Budget the **whole lifecycle** of any weight change: data refresh, retrain, re-eval, redeploy. - Stabilize the task + eval **before** distilling. **Don't** - Don't fine-tune to add facts. (Use RAG.) - Don't use RAG to fix tone/format. (Use prompt or fine-tune.) - Don't fine-tune before prompt + RAG are genuinely exhausted against a real eval. - Don't fine-tune without a held-out eval set — you can't tell improvement from regression. - Don't let train and eval data overlap. Verify zero leakage. - Don't distill an unstable task — the student is obsolete on arrival. - Don't keep paying a giant repeated prompt every call when fine-tuning would bake it in. - Don't assume "one training run." Assume a retrain treadmill and price it in. --- **Cross-references:** retrieval/chunking/index/retrieval-eval detail → **`rag`** skill. Prompt structure, few-shot design, output schemas → **`prompt-engineer`** skill. Eval-suite construction → your evaluation skill of record.