Develop "quizzes" (evals) to measure model performance on specific tasks. Use these benchmarks to guide fine-tuning, determine product UX patterns, and track performance improvements over time. Use this when launching a new AI feature, switching between model versions, or optimizing for high-stakes accuracy.
SkillScore70Maintained
Automatic
◆AI Engineer pack
Last verified 9/3/2026
What it changes
Passed 16 of 23 test cases with the skill, and 9 of 23 without it.
Without skill39%
With skill70%
Checks it passed that the model alone missed
Recommends creating dedicated evaluations for each sub-task.
Recommends building for the next model iteration's capabilities.
Recommends fine-tuning with at least 1,000 problem-to-good-answer examples.
and 5 more in the full report
7 fixed9 passed either way7 still failing0 broke
Pass-rate lift
+30 pp
Tokens
+13%
Agent turns
0%
Response time
-16%
Model gemini-3.6-flashMeasured 9/3/2026v116 of 23 cases passed with skill
Tokens, turns and time describe resource use, not a dollar price.
See it in action
One of 7 test cases the skill fixed. Same model, same prompt, with and without the skill.
1 of 7
Prompt
We are establishing a hill-climbing development cycle for an enterprise AI tool. What sequential steps constitute the hill-climbing evaluation loop from initial baseline testing to task decomposition?
What was checked, without and with the skill
Checked
Without skill
With skill
Outlines hill-climbing steps including baseline evaluation, fine-tuning on problem-answer pairs, re-testing, and sub-task decomposition.
Missed
Passed
case-21 · gemini-3.6-flash · 9/3/2026 · answers are capped and redacted
What's inside
The instructions your model reads when the skill loads.
SKILL.md51 linessingle file
AI Eval Design and Iteration
In traditional software, inputs and outputs are defined. In AI, inputs and outputs are fuzzy. Evals (evaluations) are the "unit tests" for AI products. They allow you to move from "vibes-based" development to metric-driven iteration. By building a rigorous "quiz" for your model, you can determine exactly how capable your product is and where it requires human-in-the-loop scaffolding.
The Eval Workflow
1. Identify "Hero Use Cases"
Don't start with generic benchmarks (like MMLU). Instead, define the specific "hero" scenarios your product must master.
Identify the 10–20 most common or high-value queries users will give your model.
For each query, define what a "Perfect/Gold" answer looks like.
Include edge cases where you expect the model to struggle (e.g., complex reasoning or specific formatting).
2. Design the "Quiz" (The Eval)
Create a set of tests to gauge how well the model knows the subject material.
Input: The specific prompt or instruction.
Reference: The "Gold" standard answer or a set of criteria (e.g., "Must mention X," "Must not exceed 200 words").
Scoring Mechanism: Use a more powerful model (like O1 or GPT-4o) to grade the output of your production model based on your criteria.
3. Apply the "Hill Climbing" Process
Use the eval scores to guide your development cycle.
Run the eval on your baseline model.
Fine-Tune: If scores are low, provide 1,000+ examples of "Problem -> Good Answer" to the model to "teach" it the specific task.
Re-Test: Run the eval again to see if performance increased.
Iterate: If performance plateaus, break the problem down into smaller tasks (ensembling) and create specific evals for each sub-task.
4. Determine UX Based on Accuracy Thresholds
The "score" of your eval dictates the product's user interface. Kevin Weil's 60/95/99 Rule:
60% Accuracy: Build a "Co-pilot" or "Draft" experience where the user must heavily edit the output.
95% Accuracy: Build a "Human-in-the-loop" experience where the model does the work, and a human briefly reviews it.
99.5% Accuracy: Build an "Agentic" or "Automated" experience where the model acts autonomously.
Examples
Example 1: Deep Research Tool
Context: Building a tool that researches a topic for 30 minutes and writes a 20-page report.
The Eval: A prompt asking to "Compare the competitive landscape of fusion energy companies in 2024."
Criteria: Does it mention Helion? Does it cite sources? Is the report 15+ pages?
Application: If the model gets the history right but misses current news, the team adds an eval specifically for "Recency" and fine-tunes the browsing tool.