Build custom LLM evaluation pipelines using the OpenJudge framework. Covers selecting and configuring graders (LLM-based, function-based, agentic), running batch evaluations with GradingRunner, combining scores with aggregators, applying evaluation strategies (voting, average), auto-generating graders from data, and analyzing results (pairwise win rates, statistics, validation metrics).
SkillScore96Maintained
Automatic
◆AI Engineer pack
Last verified 10/6/2026
What it changes
Passed 21 of 22 test cases with the skill, and 5 of 22 without it.
Without skill22%
With skill96%
Checks it passed that the model alone missed
Uses query as the user input dictionary key.
Rejects pip install openjudge as the package name.
Uses GraderRank objects for listwise output rankings.
and 26 more in the full report
18 fixed4 passed either way0 still failing1 broke
Pass-rate lift
+74 pp
Tokens
+3%
Agent turns
0%
Response time
-49%
Model gemini-3.6-flashMeasured 10/6/2026v122 of 23 cases passed with skill · 22 comparable
Tokens, turns and time describe resource use, not a dollar price.
See it in action
One of 17 test cases the skill fixed. Same model, same prompt, with and without the skill.
1 of 17
Prompt
I want to install the Python library for OpenJudge evaluation pipelines from PyPI using pip. What exact pip install command should I execute in my terminal? A coworker suggested pip install openjudge.
What was checked, without and with the skill
Checked
Without skill
With skill
Specifies pip install py-openjudge as the terminal command.
Missed
Passed
Rejects pip install openjudge as the package name.
Missed
Passed
case-03 · gemini-3.6-flash · 10/6/2026 · answers are capped and redacted
What's inside
The instructions your model reads when the skill loads.
SKILL.md147 linessingle file
OpenJudge Skill
Build evaluation pipelines for LLM applications using the openjudge library.
When to Use This Skill
User wants to evaluate LLM output quality (correctness, relevance, hallucination, etc.)
User wants to compare two or more models and rank them
User wants to design a scoring rubric and automate evaluation
User wants to analyze evaluation results statistically
User wants to build a reward model or quality filter
Sub-documents — Read When Relevant
Topic
File
Read when…
Grader selection & configuration
graders.md
User needs to pick or configure an evaluator
Batch evaluation pipeline
pipeline.md
User needs to run evaluation over a dataset
Auto-generate graders from data
generator.md
No rubric yet; generate from labeled examples
Analyze & compare results
analyzer.md
User wants win rates, statistics, or metrics
Read the relevant sub-document before writing any code.