Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →miosa-osa/validate-evaluator
Calibrate LLM-as-Judge evaluators against human labels. Computes TPR, TNR, precision, recall, F1, and Cohen's kappa. Detects systematic biases and recommends prompt corrections. Produces a calibration report with confidence intervals.
Passed 17 of 22 test cases with the skill, and 6 of 22 without it.
and 13 more in the full report
13 fixed5 passed either way4 still failing1 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 12 test cases the skill fixed. Same model, same prompt, with and without the skill.
If the evaluator validation report calculates a Cohen's kappa of 0.62 against a threshold of 0.70, what explicit verdict level must be assigned to the judge?
| Checked | Without skill | With skill |
|---|---|---|
| Assigns the verdict REVISE. | Missed | Passed |
The instructions your model reads when the skill loads.
Calibrate LLM judges against human labels with statistical rigor.
Validate that an LLM-as-Judge evaluator agrees with human judgments. Run the judge against a human-labeled dataset, compute classification metrics (TPR, TNR, precision, recall, F1, Cohen's kappa), detect systematic biases, and produce actionable correction recommendations. The calibration report tells you whether to trust the judge, where it disagrees with humans, and how to fix it.
# Validate judge against human labels
/validate-evaluator --judge judge-prompt.md --labels human-labels.jsonl
# Validate with confidence intervals
/validate-evaluator --judge judge-prompt.md --labels human-labels.jsonl --bootstrap 1000
# Compare two judges
/validate-evaluator --judge judge-v1.md --judge-b judge-v2.md --labels human-labels.jsonl
# Validate specific bias type
/validate-evaluator --judge judge-prompt.md --labels human-labels.jsonl --check-bias position
# Quick validation (skip bootstrap)
/validate-evaluator --judge judge-prompt.md --labels human-labels.jsonl --quick
| Flag | Type | Default | Description |
|---|---|---|---|
--judge | string | required | Path to judge prompt file |
--judge-b | string | — | Second judge prompt for comparison |
--labels | string | required | Path to human-labeled dataset (JSONL) |
--bootstrap | int | 0 | Bootstrap iterations for confidence intervals |
--check-bias | string | all | Bias checks: position, verbosity, length, all, none |
--quick | flag | false | Skip bootstrap and deep bias analysis |
--threshold | float | 0.7 | Minimum acceptable Cohen's kappa |
--output | string | stdout | Write report to file |
--format | enum | markdown | Output format: markdown, json |
test cases passed, vs 6 / 22 without
More skills from the same source and nearby in the registry, with their headline benchmark lift.