LLM Model Selection — Consult Benchmarks Before Choosing

When a task requires you to recommend or select an LLM for a specific use case, follow a systematic process before defaulting to a well-known model name:

1. Check Existing Benchmarks First

If you or your team have already run benchmarks (bake-offs) for similar use cases, that prior data is more valuable than generic leaderboards. Reasons:

• Context-specific results trump averages. A model that ranks #2 on MMLU may rank #1 on your specific task (code generation, legal analysis, math reasoning).
• Real-world performance differs from benchmark suites. Generic leaderboards test broad knowledge; your task has specific constraints (latency budget, cost per call, long-context handling, instruction-following precision).
• You avoid redundant work. If you've already benchmarked "agent reasoning on financial data," reuse that result instead of re-running it on a similar financial-domain task.

If an existing benchmark covers your use case: use the winner as your default choice. Note the selection criteria so you can explain the decision.

2. When to Re-Benchmark

If your use case diverges from existing benchmarks, narrow the test scope and run a fresh benchmark:

• Scope mismatch: existing benchmark was for "summarization" but your task is "multi-step reasoning with chain-of-thought"
• Different constraints: existing test was on 4K-token documents; you need 100K-token handling
• New model release: new frontier models are available since the last benchmark
• Cost/latency window changed: you now have stricter latency budget or cost ceiling

When re-benchmarking, use existing benchmarks as prior signals — especially for known weaknesses (e.g., "GPT models historically fail at precise JSON output; Claude handles it better"). This lets you design a test that's smaller and faster than the original.

3. Always Validate Availability & Pricing Live

Never trust model names or pricing from memory or old docs. Models are deprecated, renamed, and repriced frequently. Always query the live API or vendor's current listing:

• Check the model catalog (OpenRouter, Claude API, Together.ai, local LLM registry, etc.) for current slugs and availability.
• Validate pricing — per-1M-token costs change, some models graduate from beta pricing.
• Confirm context window — older docs may list smaller limits than current releases.
• Check availability in your region — some models are geo-gated.

This query takes seconds and prevents selecting a model that no longer exists or is no longer cost-effective for your budget.

4. Generic Leaderboards as a Last Resort

If you have no prior benchmarks and no time to run one, generic leaderboards (Artificial Analysis, Vellum, HELM, MMLU rankings) are a starting point, not the final answer. Use them to identify promising candidates, then ask yourself:

• Does this leaderboard test my actual constraints? (e.g., a leaderboard for "general knowledge" doesn't predict performance on specialized code generation)
• What does the test suite optimize for? (latency, reasoning depth, factuality, instruction-following?) Does it match your priorities?
• Are the costs/latencies published alongside the scores? A model that's 1% more accurate but 10× slower may be the wrong choice.

From the leaderboard's top candidates, pick 2–3 contenders and run a small focused benchmark (10–20 examples from your real task) before committing.

Anti-Patterns to Avoid

• Treating leaderboards as oracle: "Claude scores highest on MMLU, so it's best for my task." Not necessarily — your task may have different structure.
• Choosing based on brand or hype: "Everyone uses GPT-4, so use that." Popular ≠ optimal for your use case.
• Assuming past performance is current: Model versions change, pricing shifts, availability in your region may be different. Revalidate.
• Benchmarking once and never updating: New models release every few months. Revisit selection every 3–6 months if your task is mission-critical.
• Ignoring the "why" of a choice: If you can't explain why Model A beats Model B for your task (beyond "it scored higher on a generic test"), you've chosen prematurely.

Workflow Summary

1. Do you have past benchmark data for this use case? → Use the winner; document why it was chosen.
2. Does your use case differ from past benchmarks? → Design a focused benchmark for the differing part; use past results as priors.
3. Do you have time/budget to benchmark? → Run a small (10–20 example) test on top candidates from leaderboards.
4. Validate live: Check model availability, pricing, and context window on the current API before finalizing the choice.
5. Document the decision: Write down the criteria, the benchmark (if any), and the runner-up models. This helps future decisions and justifies the choice to stakeholders.

Applies To

• All AI agent frameworks and development environments (Claude Code, Cursor, etc.).
• Any task involving LLM selection, from personal projects to production systems.
• Works for API-based models (OpenRouter, Anthropic, OpenAI, Mistral, etc.) and self-hosted models (Ollama, vLLM, local quantized runs).