--- name: avizmarlon/llm-model-selection source: https://app.decimal.ai/s/avizmarlon-llm-model-selection@1/SKILL.md source_sha256: ac0567000fd8 --- # LLM Model Selection — Consult Benchmarks Before Choosing When a task requires you to recommend or select an LLM for a specific use case, follow a systematic process before defaulting to a well-known model name: ## 1. Check Existing Benchmarks First If you or your team have already run benchmarks (bake-offs) for similar use cases, that prior data is more valuable than generic leaderboards. Reasons: - **Context-specific results trump averages.** A model that ranks #2 on MMLU may rank #1 on your specific task (code generation, legal analysis, math reasoning). - **Real-world performance differs from benchmark suites.** Generic leaderboards test broad knowledge; your task has specific constraints (latency budget, cost per call, long-context handling, instruction-following precision). - **You avoid redundant work.** If you've already benchmarked "agent reasoning on financial data," reuse that result instead of re-running it on a similar financial-domain task. **If an existing benchmark covers your use case:** use the winner as your default choice. Note the selection criteria so you can explain the decision. ## 2. When to Re-Benchmark If your use case **diverges from existing benchmarks**, narrow the test scope and run a fresh benchmark: - **Scope mismatch:** existing benchmark was for "summarization" but your task is "multi-step reasoning with chain-of-thought" - **Different constraints:** existing test was on 4K-token documents; you need 100K-token handling - **New model release:** new frontier models are available since the last benchmark - **Cost/latency window changed:** you now have stricter latency budget or cost ceiling When re-benchmarking, use existing benchmarks as **prior signals** — especially for known weaknesses (e.g., "GPT models historically fail at precise JSON output; Claude handles it better"). This lets you design a test that's smaller and faster than the original. ## 3. Always Validate Availability & Pricing Live **Never trust model names or pricing from memory or old docs.** Models are deprecated, renamed, and repriced frequently. Always query the live API or vendor's current listing: - Check the **model catalog** (OpenRouter, Claude API, Together.ai, local LLM registry, etc.) for current slugs and availability. - Validate **pricing** — per-1M-token costs change, some models graduate from beta pricing. - Confirm **context window** — older docs may list smaller limits than current releases. - Check **availability in your region** — some models are geo-gated. This query takes seconds and prevents selecting a model that no longer exists or is no longer cost-effective for your budget. ## 4. Generic Leaderboards as a Last Resort If you have no prior benchmarks and no time to run one, generic leaderboards (Artificial Analysis, Vellum, HELM, MMLU rankings) are a starting point, **not the final answer.** Use them to identify **promising candidates**, then ask yourself: - **Does this leaderboard test my actual constraints?** (e.g., a leaderboard for "general knowledge" doesn't predict performance on specialized code generation) - **What does the test suite optimize for?** (latency, reasoning depth, factuality, instruction-following?) Does it match your priorities? - **Are the costs/latencies published alongside the scores?** A model that's 1% more accurate but 10× slower may be the wrong choice. From the leaderboard's top candidates, pick **2–3 contenders** and run a small focused benchmark (10–20 examples from your real task) before committing. ## Anti-Patterns to Avoid - **Treating leaderboards as oracle:** "Claude scores highest on MMLU, so it's best for my task." Not necessarily — your task may have different structure. - **Choosing based on brand or hype:** "Everyone uses GPT-4, so use that." Popular ≠ optimal for your use case. - **Assuming past performance is current:** Model versions change, pricing shifts, availability in your region may be different. Revalidate. - **Benchmarking once and never updating:** New models release every few months. Revisit selection every 3–6 months if your task is mission-critical. - **Ignoring the "why" of a choice:** If you can't explain why Model A beats Model B for *your* task (beyond "it scored higher on a generic test"), you've chosen prematurely. ## Workflow Summary 1. **Do you have past benchmark data for this use case?** → Use the winner; document why it was chosen. 2. **Does your use case differ from past benchmarks?** → Design a focused benchmark for the differing part; use past results as priors. 3. **Do you have time/budget to benchmark?** → Run a small (10–20 example) test on top candidates from leaderboards. 4. **Validate live:** Check model availability, pricing, and context window on the current API before finalizing the choice. 5. **Document the decision:** Write down the criteria, the benchmark (if any), and the runner-up models. This helps future decisions and justifies the choice to stakeholders. ## Applies To - All AI agent frameworks and development environments (Claude Code, Cursor, etc.). - Any task involving LLM selection, from personal projects to production systems. - Works for API-based models (OpenRouter, Anthropic, OpenAI, Mistral, etc.) and self-hosted models (Ollama, vLLM, local quantized runs).