Use when the user has evaluation principles or a dataset but needs help choosing the right graders, designing evaluation metrics, creating LLM-as-judge prompts, combining multiple metrics into a composite score, or building an automated evaluation pipeline. Also use when the user mentions grader selection, metric design, judge prompt engineering, rubric design, evaluation pipeline code, or "how to evaluate [X] automatically." Outputs executable OpenJudge pipeline code.
SkillScore91Maintained
Automatic
◆AI Engineer pack
Last verified 8/14/2026
What it changes
Passed 19 of 21 test cases with the skill, and 8 of 21 without it.
Without skill36%
With skill91%
Checks it passed that the model alone missed
The judge prompt includes a borderline few-shot example.
The selection uses RelevanceGrader for answer relevance.
The selection uses CorrectnessGrader for factual accuracy.
and 20 more in the full report
12 fixed8 passed either way2 still failing0 broke
Pass-rate lift
+55 pp
Tokens
+182%
Agent turns
0%
Response time
-14%
Model gemini-3.6-flashMeasured 8/14/2026v120 of 22 cases passed with skill · 21 comparable
Tokens, turns and time describe resource use, not a dollar price.
See it in action
One of 11 test cases the skill fixed. Same model, same prompt, with and without the skill.
1 of 11
Prompt
We need an evaluation prompt for an LLM judge that reviews AI customer service responses to check if mandatory health disclaimers are correctly included. Please generate a structured judge prompt that includes clear pass/fail criteria, realistic examples (including tricky cases), and forces a structured JSON output with reasoning prior to the final decision.
What was checked, without and with the skill
Checked
Without skill
With skill
The response wraps the evaluation prompt inside an OpenJudge LLMGrader instance.
Missed
Passed
The prompt output JSON schema specifies a critique key before the result key.
Passed
Passed
The prompt output schema uses binary Pass or Fail values rather than a numerical scale.
Passed
Passed
case-01 · gemini-3.6-flash · 8/14/2026 · answers are capped and redacted
What's inside
The instructions your model reads when the skill loads.
SKILL.md487 linessingle file
Metric Design
Select, configure, and combine evaluation graders into a working pipeline. You choose
the right tool for each evaluation dimension — from zero-cost code checks to LLM judges
— and produce executable GradingRunner code that runs on OpenJudge.
Requires OpenJudge (pip install py-openjudge). This skill is intentionally
SDK-centric — grader selection, GradingRunner, and aggregators are OpenJudge APIs. The
design/decision logic still applies if you use another harness; only the code does not.
When to Activate
User has eval dimensions/principles but doesn't know which grader type to use
User wants to write an LLM-as-judge prompt for a specific failure mode
User needs a composite score combining multiple evaluation dimensions
User wants to auto-generate graders from labeled data instead of writing them manually
User's current evaluation is all LLM-based and too expensive/too slow
Checklist
You MUST create a task for each item and complete them in order:
Select grader types — per dimension, pick the right grader class
Create custom graders — write judge prompts (4-component) or function graders
Auto-generate if applicable — use OpenJudge Generator for cold starts
Run anti-pattern scan — check for Likert, missing few-shot, vague criteria
Build pipeline code — assemble GradingRunner with graders + aggregators
Step 1: Select Grader Type Per Dimension
For each evaluation dimension, walk this decision tree (first match wins):
1. Can a deterministic rule check this?
→ StringMatchGrader / JsonValidatorGrader / FunctionGrader (zero cost, 100% consistent)
Examples: exact match for classification labels, regex for format checks,
JSON schema validation, keyword presence/absence
2. Does it require semantic understanding of text quality?
→ LLMGrader with built-in class (low cost, pre-optimized)