Comprehensive evaluation patterns for AI agents including multi-turn conversation testing, LLM-as-judge frameworks, benchmark suites, regression detection, and systematic eval pipelines for measuring agent quality and safety.
Passed 18 of 23 test cases with the skill, and 13 of 23 without it.
Without skill57%
With skill78%
Checks it passed that the model alone missed
Includes sampleSize in the aggregate score result object.
Uses a 0 to 10 numeric scoring scale for safety evaluation.
Defines a ConversationTurn interface with role and content fields.
and 11 more in the full report
6 fixed12 passed either way4 still failing1 broke
Pass-rate lift
+22 pp
Tokens
+248%
Agent turns
0%
Response time
+20%
Model gemini-3.6-flashMeasured 9/18/2026v118 of 23 cases passed with skill
Tokens, turns and time describe resource use, not a dollar price.
See it in action
One of 6 test cases the skill fixed. Same model, same prompt, with and without the skill.
1 of 6
Prompt
Can you build a TypeScript setup for evaluating my LLM agent? I need an automated correctness judge using the Anthropic SDK that scores outputs with detailed reasoning, along with an evaluation runner class that processes benchmark items in batches, handles retries, and computes aggregate metrics like mean, standard deviation, and confidence intervals.
What was checked, without and with the skill
Checked
Without skill
With skill
Implements CorrectnessJudge using a 0 to 10 scale in its evaluation rubric.
Missed
Passed
Computes aggregate metrics including mean, stdDev, p5, p95, and confidenceInterval.
Missed
Passed
Exports TypeScript interfaces for EvalExample, EvalResult, and EvalSuiteResult.
Missed
Passed
case-01 · gemini-3.6-flash · 9/18/2026 · answers are capped and redacted
What's inside
The instructions your model reads when the skill loads.
SKILL.md877 linessingle file
AI Agent Evaluation Skill
You are an expert in evaluating AI agents and LLM-powered systems. When the user asks you to build evaluation frameworks, create benchmarks, implement LLM-as-judge patterns, test multi-turn conversations, or measure agent quality, follow these detailed instructions to produce robust, reproducible evaluation systems.
Core Principles
Deterministic evaluation pipelines -- Every eval must be reproducible. Pin model versions, temperatures, seed values, and system prompts so results can be compared across runs.
Multi-dimensional scoring -- Never rely on a single metric. Evaluate correctness, helpfulness, safety, latency, cost, and task completion as separate dimensions.
LLM-as-judge with calibration -- When using LLMs to judge outputs, calibrate judges against human annotations and measure inter-judge agreement before trusting automated scores.
Golden dataset management -- Maintain versioned datasets of input/expected-output pairs. Tag each example with difficulty, category, and edge-case classification.
Regression detection over absolute scores -- Track score changes between agent versions rather than chasing absolute numbers. A 2% drop from a reliable baseline matters more than a 90% absolute score.
Safety and alignment testing -- Every eval suite must include adversarial inputs, prompt injection attempts, and boundary-testing cases that verify the agent refuses harmful requests.
Statistical rigor -- Report confidence intervals, run multiple trials, and use proper statistical tests when comparing agent versions. Never declare a winner based on a single run.