ai mlby agentscope-ai
OpenJudge Skill
agentscope-ai/openjudge
Build custom LLM evaluation pipelines using the OpenJudge framework. Covers selecting and configuring graders (LLM-based, function-based, agentic), running batch evaluations with GradingRunner, combining scores with aggregators, applying evaluation strategies (voting, average), auto-generating graders from data, and analyzing results (pairwise win rates, statistics, validation metrics).
What it changes
Passed 23 of 23 test cases with the skill, and 6 of 23 without it.
Without skill26%
With skill100%
Checks it passed that the model alone missed
- Executes evaluation using GradingRunner.
- Calls arun on the GradingRunner instance.
- Uses query as key for prompt in dataset items.
and 22 more in the full report
17 fixed6 passed either way0 still failing0 broke
- Pass-rate lift
- +74 pp
- Tokens
- +39%
- Agent turns
- 0%
- Response time
- -49%
Model gemini-3.6-flashMeasured 8/14/2026v123 of 23 cases passed with skill
See it in action
One of 17 test cases the skill fixed. Same model, same prompt, with and without the skill.
PromptWhat async method name on openjudge's GradingRunner is called to execute evaluation over a list of data samples?
What was checked, without and with the skill| Checked | Without skill | With skill |
|---|
| Calls arun on the GradingRunner instance. | Missed | Passed |
|---|
What's inside
The instructions your model reads when the skill loads.
SKILL.md147 linessingle file
OpenJudge Skill
Build evaluation pipelines for LLM applications using the openjudge library.
When to Use This Skill
- User wants to evaluate LLM output quality (correctness, relevance, hallucination, etc.)
- User wants to compare two or more models and rank them
- User wants to design a scoring rubric and automate evaluation
- User wants to analyze evaluation results statistically
- User wants to build a reward model or quality filter
Sub-documents — Read When Relevant
Read the relevant sub-document before writing any code.
Install
pip install py-openjudge
Architecture Overview
Dataset (List[dict])
│
▼
GradingRunner ← orchestrates everything
│
├─► Grader A ──► EvaluationStrategy ──► _aevaluate() ──► GraderScore / GraderRank
├─► Grader B ──► EvaluationStrategy ──► _aevaluate() ──► GraderScore / GraderRank
└─► Grader C ...
Track record
Benchmark
23 / 23
test cases passed, vs 6 / 23 without
Adoption
- Users
- 0
- Times used
- 0
Safety and source details
SkillSafety
✓ 3 of 3 passed✓Code scan15 of 15 checks clean
0 critical · 0 warning · 0 info — scanner v4 — 8/3/2026
View 15 scanner checks
live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing
✓IntentSafe
LLM judge · does the skill do what it claims
✓ContentClean
LLM judge · policy & harmful-content review
Source
ImportedLicense Apache-2.0Imported 8/3/2026Version v1