Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →miosa-osa/judge-prompt
Design binary pass/fail LLM-as-Judge evaluators. Structured prompt engineering for evaluation: criteria definition, rubric construction, few-shot calibration, and bias mitigation. Produces a ready-to-deploy judge prompt with scoring instructions.
Passed 21 of 22 test cases with the skill, and 10 of 22 without it.
and 10 more in the full report
11 fixed10 passed either way1 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 11 test cases the skill fixed. Same model, same prompt, with and without the skill.
If we run `/judge-prompt --task "translation fidelity"` without passing `--format` or `--output`, what file extension and file name will the resulting prompt artifact have?
| Checked | Without skill | With skill |
|---|---|---|
| Specifies judge-prompt.md as the default output filename. | Missed | Passed |
The instructions your model reads when the skill loads.
Design binary pass/fail LLM-as-Judge evaluation prompts.
Create a rigorous LLM-as-Judge prompt that evaluates model outputs with binary pass/fail decisions. Walks through criteria definition, rubric construction, few-shot example selection, bias mitigation, and prompt assembly. The output is a self-contained judge prompt ready for deployment in an eval pipeline, with built-in guardrails against common judge biases (position, verbosity, self-preference).
# Interactive judge design
/judge-prompt --task "summarization quality"
# From criteria spec
/judge-prompt --criteria criteria.yaml --examples examples.jsonl
# Design judge for specific eval
/judge-prompt --eval evals/code-review/ --task "code correctness"
# Add bias mitigation
/judge-prompt --task "helpfulness" --mitigate position,verbosity
# Generate judge with calibration examples
/judge-prompt --task "factual accuracy" --calibrate labels/human-ratings.jsonl
| Flag | Type | Default | Description |
|---|---|---|---|
--task | string | required | What the judge evaluates (e.g., "summarization quality") |
--criteria | string | — | Path to YAML criteria specification |
--examples | string | — | Path to JSONL file with labeled examples for few-shot |
--eval | string | — | Path to eval directory for context |
--mitigate | string | all | Biases to mitigate: position, verbosity, self-preference, all, none |
--calibrate | string | — | Human labels file for calibration examples |
--output | string | judge-prompt.md | Output path for the judge prompt |
--format | enum | markdown | Output format: markdown, yaml, json |
--style | enum | binary | Judgment style: binary (pass/fail), likert (1-5), comparative (A vs B) |
test cases passed, vs 10 / 22 without
More skills from the same source and nearby in the registry, with their headline benchmark lift.