Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →growthxai/output-eval-validate-judge
Validate LLM judges against human labels using TPR/TNR metrics and train/dev/test splits. Use after writing a judge prompt to verify it agrees with human judgment.
Passed 22 of 23 test cases with the skill, and 12 of 23 without it.
and 12 more in the full report
11 fixed11 passed either way0 still failing1 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 11 test cases the skill fixed. Same model, same prompt, with and without the skill.
When selecting few-shot examples to insert into an LLM evaluator prompt template during prompt engineering, which dataset split can examples be drawn from?
| Checked | Without skill | With skill |
|---|---|---|
| Restricts few-shot selection exclusively to the train split | Missed | Passed |
The instructions your model reads when the skill loads.
An LLM judge is only useful if it agrees with human judgment. This skill walks you through calibrating a judge against human-labeled data using True Positive Rate (TPR) and True Negative Rate (TNR) metrics. Do this before trusting any judgeVerdict(), judgeScore(), or judgeLabel() evaluator in your eval suite.
.prompt file — Written following output-eval-judge-promptground_truth.evals.<evaluator_name>.verdict: pass or failThis process applies only to LLM-based judges. For code-based Verdict.* evaluators, write unit tests instead.
Split your labeled datasets into three groups:
| Split | % of Data | Purpose | Example (100 datasets) |
|---|---|---|---|
| Train | 10-20% | Source of few-shot examples in the judge prompt | 15 datasets |
| Dev | 40-45% | Iterate on judge prompt, measure TPR/TNR | 42 datasets |
| Test | 40-45% | Final held-out measurement, run once | 43 datasets |
Use a naming convention or subdirectories to separate splits:
Option A: Name prefixes
tests/datasets/
├── train_formal_pass_01.yml
├── train_casual_fail_01.yml
├── dev_technical_pass_01.yml
├── dev_ambiguous_fail_01.yml
├── test_simple_pass_01.yml
├── test_contradictory_fail_01.yml
└── ...
test cases passed, vs 12 / 23 without
More skills from the same source and nearby in the registry, with their headline benchmark lift.