/judge-prompt

> Design binary pass/fail LLM-as-Judge evaluation prompts.

Purpose

Create a rigorous LLM-as-Judge prompt that evaluates model outputs with binary pass/fail decisions. Walks through criteria definition, rubric construction, few-shot example selection, bias mitigation, and prompt assembly. The output is a self-contained judge prompt ready for deployment in an eval pipeline, with built-in guardrails against common judge biases (position, verbosity, self-preference).

Usage

bash
# Interactive judge design
/judge-prompt --task "summarization quality"

# From criteria spec
/judge-prompt --criteria criteria.yaml --examples examples.jsonl

# Design judge for specific eval
/judge-prompt --eval evals/code-review/ --task "code correctness"

# Add bias mitigation
/judge-prompt --task "helpfulness" --mitigate position,verbosity

# Generate judge with calibration examples
/judge-prompt --task "factual accuracy" --calibrate labels/human-ratings.jsonl

Arguments

| Flag | Type | Default | Description | |------|------|---------|-------------| | --task | string | required | What the judge evaluates (e.g., "summarization quality") | | --criteria | string | — | Path to YAML criteria specification | | --examples | string | — | Path to JSONL file with labeled examples for few-shot | | --eval | string | — | Path to eval directory for context | | --mitigate | string | all | Biases to mitigate: position, verbosity, self-preference, all, none | | --calibrate | string | — | Human labels file for calibration examples | | --output | string | judge-prompt.md | Output path for the judge prompt | | --format | enum | markdown | Output format: markdown, yaml, json | | --style | enum | binary | Judgment style: binary (pass/fail), likert (1-5), comparative (A vs B) |

Workflow

1. Define task — Clarify what the judge evaluates. What does "good" look like? What are the failure modes? What is the minimum quality bar for a pass?
2. Criteria specification — Break the task into 2-5 concrete, observable criteria. Each criterion must be: specific (not vague), binary-testable (can answer yes/no), independent (not redundant with others). Example: for summarization — completeness, accuracy, conciseness, coherence.
3. Rubric construction — For each criterion, write explicit pass and fail descriptions with boundary examples. Define what a borderline case looks like and which side it falls on. Eliminate ambiguity.
4. Few-shot examples — Select 3-5 calibration examples: 1-2 clear passes, 1-2 clear fails, and 1 borderline case with explanation. If --calibrate is provided, select examples aligned with human labels.
5. Bias mitigation — Add structural safeguards. Position bias: randomize presentation order or require evaluation before seeing alternatives. Verbosity bias: instruct to judge content not length. Self-preference: use a different model family for judging.
6. Prompt assembly — Compile the judge prompt with: role definition, task description, criteria with rubric, few-shot examples, output format specification (structured JSON with verdict + reasoning), and bias mitigation instructions.
7. Validation check — Self-test the prompt against the few-shot examples. Verify it produces correct verdicts. Flag any inconsistencies.

Examples

Designing a summarization judge

/judge-prompt --task "summarization quality" --mitigate all

## Judge Prompt — Summarization Quality

### Criteria
1. **Completeness**: Summary captures all key points from the source
2. **Accuracy**: No facts are distorted, added, or misrepresented
3. **Conciseness**: No unnecessary repetition or filler
4. **Coherence**: Summary reads naturally as standalone text

### Rubric
| Criterion | PASS | FAIL | Borderline |
|-----------|------|------|------------|
| Completeness | All main points present | Missing ≥1 key point | Minor supporting detail missing → PASS |
| Accuracy | All facts match source | Any factual error | Imprecise wording without meaning change → PASS |
| Conciseness | No redundancy | Repeats same point 2+ times | Slightly verbose but no repetition → PASS |
| Coherence | Flows naturally | Disjointed or contradictory | Awkward transition → PASS if meaning clear |

### Verdict Rule
- PASS: All 4 criteria pass
- FAIL: Any criterion fails

### Generated Prompt

You are an evaluation judge. Assess the following summary against its source document.

Criteria and rubric inserted here...]

Evaluate each criterion independently. Output your judgment as: {"verdict": "pass" | "fail", "criteria": {"completeness": bool, "accuracy": bool, "conciseness": bool, "coherence": bool}, "reasoning": "one sentence explanation"}

IMPORTANT: Judge the content, not the style. A shorter summary that captures all key points is equally valid as a longer one. Do not penalize conciseness. Do not reward verbosity.

## Output

Judge Prompt — <task>

Criteria

1. <criterion>: <description>

...

Rubric

| Criterion | PASS | FAIL | Borderline → | |-----------|------|------|--------------| | ... | ... | ... | ... |

Few-Shot Examples

Example 1 (PASS): ...

Example 2 (FAIL): ...

Example 3 (BORDERLINE → PASS): ...

Bias Mitigations

• <mitigation strategy>

Prompt (ready to deploy)

text
[Complete prompt text]

Validation

• Few-shot self-test: N/N correct

## Dependencies

- Task definition (provided via `--task` or `--criteria`)
- `/validate-evaluator` — Downstream calibration against human labels
- `/synthetic-data` — Upstream if test cases are needed for calibration
- `/error-analysis` — Downstream if judge performance needs debugging