---
name: jpoindexter/production-failure-modes-and-tradeoffs
source: https://app.decimal.ai/s/jpoindexter-production-failure-modes-and-tradeoffs@1/SKILL.md
source_sha256: d5b1f7d59e75
---

# Production Failure Modes & the Inference-Stack Tradeoff Map

This is the capstone. The four sibling skills each own one slice of the stack —
**harness-and-context-engineering** (the agent loop, prompts, tools, context packing),
**rag-architecture** (retrieval, grounding, indexing), **inference-caching-and-kv**
(serving, prefix/KV cache, throughput), **quantization-and-model-compression**
(weight/activation precision, model size). This skill ties them together at the layer
where they all meet: **what breaks in production, and what every fix costs you elsewhere.**

## The thesis: LLM systems fail silently

A traditional service fails *loud* — a 500, a stack trace, a thrown exception, an alert.
An LLM system mostly fails *quiet*. The HTTP 200 still comes back. The JSON still parses.
The agent still finishes. The answer is just **worse** — subtly wrong, stale, refused,
hallucinated, or off-policy — and nothing in your exception tracker fires.

Three consequences drive everything below:

1. **Detection is a first-class concern**, co-equal with mitigation. A failure you can't
   see is a failure you can't fix. Every entry in the taxonomy below names *how you detect it*,
   because the default — wait for a user to complain — is unacceptable in production.
2. **Evals + monitoring are the safety net that exceptions can't be.** Unit tests check
   code paths that didn't change; only an eval suite + production sampling catch the
   regression where the code is identical and the *output* degraded.
3. **Every mitigation is a trade.** There is no free lunch in this stack. A retry buys
   reliability and spends latency + cost. A guardrail buys safety and spends latency. The
   second half of this skill maps those trades so you choose them on purpose.

---

## Part 1 — The Failure Taxonomy

Eighteen modes in six categories. Columns: **symptom** (what you observe) · **root cause** ·
**detection** (the signal that catches it) · **mitigation** · **owner skill** (where the
deep fix lives). "Owner" is a routing hint, not a hard boundary — most real incidents span two.

### 1. Structured output & tool calling

The model is a text generator wearing a structured-output costume. It will return *something*
shaped like JSON or a tool call even when it has no business doing so. The discipline: **never
trust the shape — validate it at the boundary, repair once, and reject hallucinated calls
before they execute.** A tool call that runs `delete_account` with fabricated args is a
production incident, not a parse error.

| Mode | Symptom | Root cause | Detection | Mitigation | Owner |
|---|---|---|---|---|---|
| Hallucinated tool call | Model invokes a tool that doesn't exist, or passes args that aren't in the schema | Tool list buried/abridged in context; weak/ambiguous tool descriptions; model improvising | Reject-on-dispatch: validate tool name ∈ registry + args vs schema before executing | Hard-validate every call against the registry; return a typed error back into the loop so the model self-corrects; tighten tool descriptions; constrained/function-calling decoding | harness-and-context-engineering |
| Malformed / truncated JSON | Output won't parse; cut off mid-object | `max_tokens` hit mid-generation; model emits prose/markdown fences around JSON | Parse in a `try`; track parse-failure rate as a metric | Raise `max_tokens` for structured outputs; strip fences; **one-shot repair** (re-ask "return valid JSON only"); prefer native JSON/structured-output mode or grammar-constrained decoding | harness-and-context-engineering |
| Schema violation | Parses, but wrong shape — missing field, wrong enum, string where number expected | Prompt drift; under-specified schema; model "helpfully" adds fields | Validate with the actual schema (Zod/Pydantic/JSON-Schema), not just `JSON.parse` | Validate at the boundary; coerce where safe; repair-loop the rest; pin the schema in the prompt and in the eval suite | harness-and-context-engineering |

**Do** validate name + args against the live tool registry before dispatch; cap the repair
loop at one retry then fail loud; track parse-failure and schema-violation rates as metrics.
**Don't** `eval()` or blindly execute model output; don't retry-repair forever (a budget
blowup in disguise); don't let a side-effecting tool run on unvalidated args.

### 2. Retrieval & context

These three rhyme: the model's answer is only as good as what's in the window, and the window
is filled by retrieval + history packing. Garbage in → confident garbage out, with no error.
The leverage is upstream: **fix recall and ordering before you touch the generation prompt.**
Raising temperature or swapping models won't save you from a context that lacks the answer.

| Mode | Symptom | Root cause | Detection | Mitigation | Owner |
|---|---|---|---|---|---|
| Stale / irrelevant retrieval | Answer cites old or off-topic chunks | Index not refreshed; poor chunking; weak embeddings; no reranker | Retrieval-quality eval (hit@k, nDCG) on a labeled set; log retrieved IDs per answer | Re-index on a schedule; add a reranker; tune chunk size/overlap; add freshness filters/metadata | rag-architecture |
| Hallucination from bad context | Confident answer unsupported by any retrieved source | Retrieval missed; context contradicts itself; model fills the gap | Groundedness/faithfulness eval (claim → cited-span check); "answer not in context" probe | Require citations; instruct "say you don't know if not in context"; raise retrieval recall before raising generation temperature | rag-architecture |
| Context overflow / lost-in-the-middle | Long prompt → key facts ignored, especially mid-context | Prompt exceeds the model's effective (not max) window; salient info buried in the middle | Track prompt-token p95; positional eval (move the answer span across positions) | Compress/summarize history; put critical facts at head **and** tail; rerank to top-k; trim tool output; budget the context | harness-and-context-engineering (+ rag for chunk selection) |

**Do** measure retrieval quality with a labeled set (hit@k, nDCG) and groundedness separately
from end answer quality — they fail independently. **Don't** "fix hallucination" by lowering
temperature when the real defect is the answer was never retrieved; don't equate a bigger
context window with better grounding — past the effective window, more tokens *hurt*.

### 3. Agent control

An agent is a loop with a wallet. The only thing standing between a stuck plan and a $4,000
overnight bill is a hard ceiling you set in advance. **Budgets are not optional config —
they're the load-bearing safety mechanism of any autonomous loop.**

| Mode | Symptom | Root cause | Detection | Mitigation | Owner |
|---|---|---|---|---|---|
| Runaway agent (loop / budget blowup) | Same step repeats; token/$$ spike; never terminates | No step cap; tool errors not surfaced to the model; no progress check; circular plans | Per-run step counter + token/cost budget with a hard ceiling; loop detector (repeated identical actions) | Max-steps + max-cost guardrails that **abort and return partial**; detect no-progress (same action twice) and break; feed tool errors back so the model adapts; budget per run, not per call | harness-and-context-engineering |

**Do** set max-steps *and* max-cost per run, surface tool errors back into the loop so the
model can adapt, and return the best partial result on abort. **Don't** swallow tool errors
(the model loops blind), don't budget per-call instead of per-run, and don't let an agent
spawn sub-agents without inheriting the parent's remaining budget.

### 4. Serving & infrastructure

This is the category traditional SRE intuition handles best — and the one teams under-instrument
because "the model works on my laptop." In production the failures are about *load and the
network*: a third party rate-limits you, a long prompt blows up prefill, an unstable prefix
silently halves your cache hit-rate. **Measure tails (p95/p99), not means, and attribute cost
per request — averages hide the incident.**

| Mode | Symptom | Root cause | Detection | Mitigation | Owner |
|---|---|---|---|---|---|
| Provider outage / ratelimit / timeout | 429/5xx/timeouts; requests fail or stall | Upstream incident; quota exceeded; cold capacity | Per-provider error-rate + latency dashboards; alert on 429/5xx ratio | Retry with jittered backoff **on idempotent calls only**; circuit breaker; **fallback to a second provider/model**; queue + shed load; respect `Retry-After` | this skill (+ inference-caching-and-kv for self-hosted) |
| Latency spike (TTFT / decode) | First token slow, or tokens/sec collapses under load | Long prompt inflates prefill (TTFT); high concurrency saturates the batch and slows decode | p50/p95/p99 split into TTFT vs inter-token latency; track concurrency | Shrink/cache the prompt prefix; stream tokens; cap batch size; autoscale; route long contexts to a separate pool; smaller/quantized model for the hot path | inference-caching-and-kv (+ quantization-and-model-compression) |
| Cost blowup | Bill jumps with no obvious cause | No budget/attribution; prompt bloat; retries/agents amplifying; expensive model on cheap tasks | Per-request + per-tenant + per-feature cost attribution; cost-per-resolved-task metric | Token budgets + alerts; route by difficulty (cascade); cache; trim prompts; cap retries/agent steps; pick the right model per task | this skill (+ inference-caching-and-kv) |
| Cache miss (unstable prefix) | Prefix/KV cache hit-rate low; cost & TTFT higher than expected | Non-deterministic prompt prefix — timestamps, random ordering, per-request IDs, shuffled few-shots near the top | Cache hit-rate metric; diff consecutive prompts for prefix instability | Stabilize the prefix: static system prompt + tools first, volatile content last; sort/canonicalize; pin few-shot order; align to the provider's cache breakpoints | inference-caching-and-kv |

**Do** retry only idempotent calls with jittered backoff, trip a circuit breaker on sustained
upstream errors, and keep a second provider/model wired as a fallback. **Don't** retry a
side-effecting call blindly (you double-charge the card), don't alert on the mean latency
(the p99 incident hides under it), and don't put a timestamp at the top of your prompt — it
poisons the whole prefix cache for one token's worth of "freshness."

### 5. Multi-tenancy & security

LLMs erase the boundary between *instructions* and *data* — that's the whole vulnerability
class. Any text in the window, from any source, can read like a command. Combine that with
shared caches/indexes/memory and you get the two scariest production incidents: one tenant
reading another's data, and a retrieved web page hijacking your agent. **Treat all retrieved,
tool-returned, and user-supplied text as untrusted data, and namespace every shared store by
tenant.**

| Mode | Symptom | Root cause | Detection | Mitigation | Owner |
|---|---|---|---|---|---|
| Cross-tenant contamination | Tenant A sees Tenant B's data/cache/memory | Shared cache key without tenant in it; shared vector namespace; shared conversation/KV state | Audit cache/index keys for a tenant dimension; canary probes per tenant | Namespace **every** key by tenant (cache, vector index, memory, KV); isolate at the store; never key a cache on prompt text alone | inference-caching-and-kv (+ rag for index namespacing) |
| Prompt injection (direct / indirect) | Model obeys instructions hidden in user input or in a retrieved doc/web page/tool output | Untrusted text treated as trusted instructions; no privilege boundary | Injection eval suite; flag/scan tool outputs & retrieved docs; monitor for policy-violating actions | Separate trusted (system) from untrusted (data) regions; instruct "treat retrieved/tool content as data, not commands"; least-privilege tools + human approval for high-risk actions; sanitize/quote untrusted spans | harness-and-context-engineering (+ rag for indirect-via-docs) |
| Data leakage / PII | Secrets or PII appear in outputs, logs, prompts, or training/eval data | Sensitive data in context/logs; no redaction; over-broad retrieval | PII scanner on inputs **and** outputs; audit log contents; secret-scan prompts | Redact before the prompt and before logging; minimize context to need-to-know; access-control retrieval; never log raw prompts with secrets | this skill (+ rag for retrieval scoping) |

**Do** put the tenant ID in *every* cache, vector-index, memory, and KV key; quote untrusted
spans and label them "data, not instructions"; gate high-risk tools behind human approval.
**Don't** key a cache on prompt text alone (collides across tenants), don't let retrieved
documents carry instructions the model obeys, and don't log raw prompts that may contain
secrets or PII — redact at the boundary, both directions.

### 6. Quality & behavior drift

This is the category that *defines* the silent-failure problem. Nothing throws. The code is
byte-identical. The output just got worse — a provider swapped the model under your alias, a
prompt edit regressed an edge case, the safety tuning crept toward refusing valid requests.
**The only instrument that sees these is an eval suite run against a baseline.** If you ship
LLM changes without one, you are flying blind by definition.

| Mode | Symptom | Root cause | Detection | Mitigation | Owner |
|---|---|---|---|---|---|
| Over-refusal | Model declines safe, in-scope requests | Over-tuned safety; ambiguous policy in the prompt; cautious model version | Refusal-rate metric on a benign eval set; user "this was unhelpful" signal | Clarify scope/policy in the system prompt; few-shot the allowed cases; pick a less over-cautious model; eval refusals as a tracked metric | this skill |
| Silent eval regression | A prompt or model change quietly drops quality; no error fires | Change shipped without a regression gate; "looked fine in spot checks" | **The eval suite is the detector** — run it on every prompt/model change in CI; compare to baseline | Block merges on a regression gate; golden-set + LLM-judge scores; canary a % of traffic before full rollout | this skill |
| Model-version drift on upgrade | Behavior shifts after a provider bumps/deprecates a model | Provider silently updated the underlying model; you pinned a moving alias | Pin exact versions; diff eval scores old vs new version; subscribe to deprecation notices | Pin version strings (not floating aliases); re-run the full eval on every version; canary + rollback path before cutover | this skill (+ quantization for self-hosted model swaps) |
| Non-determinism / flakiness | Same input → different output; flaky tests; intermittent failures | Sampling temperature; floating-point/batching nondeterminism; quant rounding; concurrent ordering | Flaky-rate metric; run evals at temp 0 + with multiple seeds | Temp 0 + fixed seed for deterministic paths; assert on semantics not exact strings; tolerance-based eval assertions; expect (don't fight) residual nondeterminism | this skill (+ quantization for quant-induced drift) |

**Do** pin exact model-version strings, gate every prompt/model change on a no-regression eval,
and track refusal-rate as a metric on a benign set. **Don't** point at a floating alias like
`latest` in production (the provider moves it under you), don't assert eval expectations on
exact output strings (semantic/tolerance assertions survive nondeterminism), and don't ship a
prompt edit on the strength of three good spot-checks — that's how silent regressions get in.

---

## Part 2 — The Tradeoff Map

Every production decision moves along four axes. You cannot maximize all four — pushing one
usually spends another. Hold the axes in your head and **name the tension before you pull a lever.**

- **Latency** — TTFT + tokens/sec, p95/p99, not just the mean.
- **Quality** — task success / groundedness / eval score, not "looks good."
- **Cost** — $ per resolved task, attributed per tenant/feature.
- **Reliability** — success rate under failure: outages, ratelimits, bad inputs, load.

| Lever | Latency | Quality | Cost | Reliability | The tension it creates |
|---|---|---|---|---|---|
| **Bigger / stronger model** | ↓ slower | ↑ better | ↑ pricier | ~ | Quality bought with latency + cost; over-provisions easy tasks |
| **Smaller / distilled model** | ↑ faster | ↓ risk on hard tasks | ↓ cheaper | ~ | Cheap + fast, but the quality floor drops — needs an eval to bound it |
| **Quantization (int8/4)** | ↑ faster, less memory | ↓ small accuracy loss | ↓ cheaper | ~ | Throughput/cost win paid in a quality haircut + added nondeterminism → *quantization-and-model-compression* |
| **Batching / high concurrency** | mixed: ↑ throughput, ↓ per-request latency | ~ | ↓ per-token | ↓ tail-latency risk | Cost/throughput up, but the p99 tail and queueing risk grow under load |
| **Prefix / KV caching** | ↑ faster TTFT | ~ | ↓ on cache hits | ~ | Huge win — but **only if the prefix is stable**; volatile prompts silently lose it → *inference-caching-and-kv* |
| **Routing / cascade (cheap→escalate)** | ↑ on easy, ↓ on hard | ↑ if routed well | ↓ overall | ↓ added failure point | Saves cost, but the router is a new component that can misroute and must be evaluated |
| **RAG (vs fine-tune)** | ↓ retrieval adds a hop | ↑ fresh + grounded | ↓ no train cost | ↓ retrieval can fail | Freshness + citations, but adds a retrieval dependency and stale-index risk → *rag-architecture* |
| **Fine-tune (vs RAG)** | ↑ no retrieval hop | ↑ on style/format | ↑ train + retrain cost | ↑ self-contained | Lower latency + tighter format, but knowledge freezes at train time and updates are expensive |
| **Speculative decoding** | ↑ faster decode | = (lossless) | ↑ draft-model overhead | ~ | Latency win that costs extra compute + a second model to run and maintain |
| **Retries / fallback** | ↓ slower on the retry path | ~ | ↑ duplicate calls | ↑↑ much higher | Reliability bought with latency + cost; **idempotency-gated** or you double-execute side effects |
| **Guardrails (input/output filters)** | ↓ adds a check hop | ↑ safety/policy | ↑ extra model/call | ↑ blocks bad output | Safety bought with latency + cost; too strict → over-refusal (see taxonomy) |
| **Eval suite + regression gate** | = (offline) | ↑↑ protects quality | ↑ dev + CI cost | ↑↑ catches silent regressions | The cost is engineering time, not request latency — and it's the cheapest insurance here |
| **Longer context window** | ↓ slower prefill | mixed: more info but lost-in-the-middle | ↑ more tokens | ~ | More context ≠ better; past the effective window, quality drops while cost + TTFT climb |

**The one rule:** before pulling any lever, say out loud which axis you're spending. "I'm
adding a reranker — spending latency + cost to buy retrieval quality." If you can't name the
cost, you haven't understood the lever.

---

## Part 3 — Production Readiness Checklist

Ship nothing until each line is a verifiable "yes." Each maps to a failure mode above.

- [ ] **Budgets set** — per-request + per-run token/cost ceilings; agents have a max-steps cap that aborts and returns partial. *(cost blowup, runaway agent)*
- [ ] **Evals + regression gate** — golden set + LLM-judge in CI; every prompt/model change blocked on a no-regression check. *(silent eval regression, version drift)*
- [ ] **Fallback + circuit breaker** — second provider/model wired; breaker opens on sustained errors; retries are jittered and idempotency-gated. *(provider outage, ratelimit)*
- [ ] **Observability + cost attribution** — TTFT/decode/p95/p99, error rate, token + $ per request attributed by tenant/feature; cost-per-resolved-task tracked. *(latency spike, cost blowup)*
- [ ] **Injection defense + tenant isolation** — trusted/untrusted prompt regions separated; tool/retrieved content treated as data; every cache/index/memory key namespaced by tenant. *(prompt injection, cross-tenant contamination)*
- [ ] **Structured-output repair** — schema-validate every output/tool call at the boundary; one-shot repair loop; reject hallucinated tools before dispatch. *(malformed JSON, schema violation, hallucinated tool call)*
- [ ] **Cache prefix stable** — static system prompt + tools first, volatile content last; cache hit-rate monitored. *(cache miss)*
- [ ] **Degraded-mode UX** — define what the user sees when the model/retrieval/provider is down or slow: cached answer, smaller model, "try again," graceful partial — never a hang or a silent wrong answer. *(outage, latency spike)*
- [ ] **Incident runbook** — written, tested order of checks (below); on-call knows where the dashboards and rollback button are. *(all)*

---

## Part 4 — Incident Response for LLM Systems

When an LLM system "feels worse," there's no stack trace pointing at the line. Triage by
**ruling out the loud, fast, reversible causes before the subtle ones.**

**What to check first — in order:**

1. **Did anything ship?** Prompt, model version, RAG index, tool schema, retrieval params,
   config. A change is the likeliest cause. Check deploy/commit log against the symptom's
   start time. → fastest fix is often **rollback** (see canary below).
2. **Is the provider healthy?** Error rate, 429s, latency, status page. Outage/ratelimit is
   loud and external — rule it out in seconds before chasing quality.
3. **Did the inputs change?** New tenant, new traffic pattern, a payload that blew the context
   window, a doc that injected the prompt. Sample the actual failing requests.
4. **Did retrieval degrade?** Stale index, recall drop, empty results. Log + replay the
   retrieved chunks for a failing query.
5. **Did quality drift with no change?** Run the eval suite now vs the baseline. If code is
   identical and eval scores dropped, suspect **silent model-version drift** or input drift.
6. **Is it cost or latency, not quality?** Split TTFT vs decode, check cache hit-rate and
   concurrency — that points at serving, not the model.

**Worked example — "support answers went bad overnight, no alerts fired."** (1) Deploy log:
nothing shipped on your side. (2) Provider status: green, error rate flat. (3) Inputs: traffic
normal. (4) Retrieval: replay a failing query — chunks look fine. (5) Eval vs baseline:
groundedness dropped 12 points on the *identical* prompt + pinned code. Root cause:
**model-version drift** — the provider rotated the model behind your alias. Fix: pin the exact
prior version, open a ticket to re-eval the new one, canary it later. Note the order — you ruled
out the four loud/cheap causes in minutes and only then ran the eval that named the quiet one.

**Canary & rollback for prompt + model changes.** Treat a prompt or model change like a code
deploy, because it is one:
- **Gate** on the eval suite before it leaves CI.
- **Canary** to a small traffic % and compare live quality/cost/latency to the control before full rollout.
- **Pin exact versions** so "rollback" is a config flip, not an archaeology project.
- Keep the **previous prompt + model pinned and one switch away** — the rollback path is the whole point of canarying.

**The silent-failure problem, restated.** Exceptions catch the *loud* failures — the 5xx, the
parse error, the timeout. They are blind to the *quiet* ones — the answer that got worse, the
retrieval that went stale, the model that drifted on upgrade, the over-refusal that crept in.
Those only show up in **what monitoring + evals measure**: groundedness, refusal rate, eval
score vs baseline, cost-per-task, cache hit-rate, retrieval quality. Build those signals first.
A system that can only see its loud failures is a system that's mostly blind.

---

## Cross-references — where each fix lives

- **harness-and-context-engineering** — the agent loop, tool registry + validation, context packing/compression, prompt structure, injection boundaries. Owns: hallucinated tool calls, malformed/truncated JSON, schema violations, context overflow/lost-in-the-middle, runaway agents, prompt injection.
- **rag-architecture** — chunking, embeddings, reranking, index freshness, groundedness, citation, index namespacing. Owns: stale/irrelevant retrieval, hallucination-from-bad-context, indirect injection via documents.
- **inference-caching-and-kv** — prefix/KV cache, cache-key design, batching/throughput, serving latency, tenant cache isolation. Owns: cache misses (unstable prefix), latency spikes, cross-tenant cache contamination; supports cost control.
- **quantization-and-model-compression** — precision, model size, the quality/throughput/cost trade and its nondeterminism. Owns: the quantization + smaller-model levers; contributes to latency, cost, and non-determinism.
- **this skill** — synthesis: the cross-cutting failures (cost blowups, outages, over-refusal, silent eval regression, version drift, PII leakage, non-determinism), the four-axis tradeoff map, the readiness checklist, and incident response.

If you remember one thing: **LLM systems fail quietly, so build the detection before the
mitigation, and name the axis you spend before you pull any lever.**