Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →jpoindexter/inference-caching-and-kv
Reference-grade guide to caching in LLM inference — provider prompt caching (Anthropic cache_control breakpoints, OpenAI automatic prefix caching, Gemini implicit/explicit), semantic caching, and KV-cache internals & management (PagedAttention/vLLM, RadixAttention/SGLang, eviction, quantized KV, memory pressure, multi-tenant safety). Concrete numbers, formulas, failure modes.
Passed 22 of 22 test cases with the skill, and 21 of 22 without it.
1 fixed21 passed either way0 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
A test case the skill fixed. Same model, same prompt, with and without the skill.
An application sends identical long prompts to OpenAI API across distributed worker nodes, but prompt cache hit rate is below 10%. Which API request parameter should be populated to ensure requests route to the same backend server family for high cache reuse?
| Checked | Without skill | With skill |
|---|---|---|
| Recommends using prompt_cache_key to route requests consistently to backend caches. | Missed | Passed |
The instructions your model reads when the skill loads.
Three distinct caches get confused under one word. They live at different layers, have different correctness properties, and fail differently.
| Cache | Layer | Match | Returns | Risk if wrong |
|---|---|---|---|---|
| Prompt cache | Provider / serving engine | Exact token-prefix | Skips prefill compute, regenerates output | None (cheaper, identical output) |
| Semantic cache | Your app, in front of the API | Embedding similarity | A stored prior response | Serves a wrong answer |
| KV cache | GPU, inside the model | Per-token K/V tensors | Enables O(1) decode | OOM / preemption |
Rule of thumb: prompt cache is exact + cheap + safe — on always. Semantic cache is fuzzy + risky — use surgically. KV cache is not optional (it's how decode works) and at scale dominates GPU memory.
The model's prefill phase computes K/V for every prompt token before the first output token, at cost O(prompt_length). Prompt caching stores that computed K/V keyed by an exact token prefix; on a hit the engine skips prefill for the cached span and resumes from the first divergent token. Matching is on exact tokens from position 0 — a prefix match, not fuzzy or substring. One changed byte invalidates everything from that byte onward. Prefill is the dominant cost for RAG/agent/long-context workloads where the prompt dwarfs the completion, so caching it turns repeated large prefixes into near-free reads.
Anthropic (explicit, cache_control breakpoints):
Break-even: a write costs 0.25× extra (5-min), each read saves 0.9×, so a cached prefix pays for itself after ~1 reuse within the TTL (0.25 < 0.9). A 50-call agent loop reusing a 20k-token system+tools prefix pays 1.25× once + 0.1× × 49
test cases passed, vs 21 / 22 without
More skills from the same source and nearby in the registry, with their headline benchmark lift.