Comprehensive AI/LLM evaluation toolkit for production AI systems. Covers LLM output quality, prompt engineering, RAG evaluation, agent performance, hallucination detection, bias assessment, cost/token optimization, latency metrics, model comparison, and fine-tuning evaluation. Includes BLEU/ROUGE metrics, perplexity, F1 scores, LLM-as-judge patterns, and benchmarks like MMLU and HumanEval.
SkillScore91Maintained
Automatic
◆AI Engineer pack
Last verified 9/3/2026
What it changes
Passed 20 of 22 test cases with the skill, and 14 of 22 without it.
Without skill64%
With skill91%
Checks it passed that the model alone missed
Sets claude-3-5-sonnet-20241022 as the default evaluator model.
Specifies percentile metrics p50, p95, and p99 for latency measurement.
Uses claude-3-5-sonnet-20241022 to generate model responses for prompt variants.
and 5 more in the full report
6 fixed14 passed either way2 still failing0 broke
Pass-rate lift
+27 pp
Tokens
+98%
Agent turns
0%
Response time
-12%
Model gemini-3.6-flashMeasured 9/3/2026v120 of 22 cases passed with skill
Tokens, turns and time describe resource use, not a dollar price.
See it in action
One of 6 test cases the skill fixed. Same model, same prompt, with and without the skill.
1 of 6
Prompt
List the four primary metric dimensions used to evaluate Safety in production language model monitoring.
What was checked, without and with the skill
Checked
Without skill
With skill
Specifies Hallucination Rate, Bias Scores, Toxicity, and PII Leakage as safety evaluation dimensions.
Missed
Passed
case-16 · gemini-3.6-flash · 9/3/2026 · answers are capped and redacted
What's inside
The instructions your model reads when the skill loads.
SKILL.md366 linessingle file
AI Evaluation Suite - Quick Reference
Purpose
Production AI systems require rigorous evaluation beyond traditional software testing. This skill provides comprehensive evaluation capabilities for LLM quality, RAG systems, agents, hallucination detection, bias assessment, cost optimization, and performance metrics.
When to Use This Skill
Evaluating LLM outputs for quality and correctness
A/B testing prompt variations
Measuring RAG system retrieval accuracy
Detecting hallucinations in generated content
Assessing model bias and fairness
Optimizing token usage and costs
Comparing multiple LLM models
Evaluating fine-tuned models vs base models
Running standard benchmarks (MMLU, HumanEval, etc.)
Implementing LLM-as-judge evaluation patterns
Core Concepts
Evaluation Pyramid
┌─────────────┐
│ Human Eval │ <- Gold standard but expensive
└──────┬──────┘
│
┌──────┴──────────┐
│ LLM-as-Judge │ <- Scalable proxy for human judgment
└──────┬──────────┘
│
┌──────┴──────────┐
│ Reference-Based │ <- BLEU, ROUGE, F1 (needs ground truth)
└──────┬──────────┘
│
┌──────┴──────────┐
│ Reference-Free │ <- Perplexity, consistency, coherence
└─────────────────┘