Use when the user has a judge/grader and human-labeled data, and wants to measure how well the judge agrees with humans, detect systematic biases, determine whether automatic evaluation can replace human review, or build a human-reduction roadmap. Also use when the user mentions calibration, TPR/TNR, judge validation, inter-rater agreement, Cohen's kappa, bias detection, or "is my automatic evaluation trustworthy." Merges the calibrate and align functions into one skill.
SkillScore88Maintained
Automatic
◆AI Engineer pack
Last verified 8/14/2026
What it changes
Passed 20 of 23 test cases with the skill, and 11 of 23 without it.
Without skill46%
With skill88%
Checks it passed that the model alone missed
Computes the net directional bias as +0.20.
Labels the disagreement cause as judge_prompt_ambiguous.
Classifies the discrepancy root cause as human_inconsistent.
and 9 more in the full report
11 fixed10 passed either way2 still failing1 broke
Pass-rate lift
+42 pp
Tokens
+122%
Agent turns
0%
Response time
-20%
Model gemini-3.6-flashMeasured 8/14/2026v121 of 24 cases passed with skill · 23 comparable
Tokens, turns and time describe resource use, not a dollar price.
See it in action
One of 10 test cases the skill fixed. Same model, same prompt, with and without the skill.
1 of 10
Prompt
During error analysis, we notice the AI judge strictly penalizes responses missing a recap section, whereas human guidelines never specified whether recaps were required. Which root cause label applies?
What was checked, without and with the skill
Checked
Without skill
With skill
Labels the disagreement cause as judge_prompt_ambiguous.
Missed
Passed
case-18 · gemini-3.6-flash · 8/14/2026 · answers are capped and redacted
What's inside
The instructions your model reads when the skill loads.
SKILL.md283 lines2 filesincludes scripts
Align Human
Measure whether your automatic judge agrees with human judgment, detect where and
why they disagree, and build a roadmap to reduce human review over time.
When to Activate
You have a working judge/grader and 50+ human-labeled examples
You want to know if the judge is trustworthy enough to replace human review
You've noticed the judge's decisions being overturned by humans
You're preparing to deploy an evaluation as a production gate
Checklist
You MUST create a task for each item and complete them in order:
Load paired data — match judge verdicts with human labels
Build human-reduction roadmap — 4-phase transition plan
Confirm and record — one confirmation, then write results
Fast path: run the bundled script
Don't hand-write the calibration statistics — that is exactly where subtle bugs hide. Run
the bundled, tested script (scripts/calibration.py, standard library only, no OpenJudge
dependency):
python scripts/calibration.py --pairs pairs.jsonl # one paired file, OR
python scripts/calibration.py --verdicts verdicts.jsonl --labels labels.jsonl --stratum-key difficulty