Designing LLM Judge Prompts

Designing LLM Judge Prompts

Overview

An LLM judge evaluates workflow output for a single, specific failure mode identified during error analysis. This skill covers how to design the .prompt file that powers judgeVerdict(), judgeScore(), or judgeLabel() calls. For the file format basics, see output-dev-prompt-file. For error analysis, see output-eval-error-analysis.

Prerequisites

Before writing a judge prompt:

1. Error analysis is complete — You have identified the specific failure mode this judge targets (from output-eval-error-analysis)
2. 20+ labeled examples — At least 20 pass and 20 fail traces for this failure mode, with ground_truth labels in dataset YAML files
3. Code-based check ruled out — Confirmed that Verdict.* helpers (contains, matches, gte, etc.) cannot reliably detect this failure

The Four Components

Every effective judge prompt has exactly four components.

1. Task and Criterion

State the single failure mode being evaluated. Be specific and observable.

Good criteria (specific, observable):

• "Does the blog post maintain a formal tone throughout, or does it slip into casual language?"
• "Does the output contain any URLs that are fabricated rather than drawn from the input?"
• "Does the summary faithfully represent the source material without adding claims not present in the original?"

Bad criteria (vague, holistic):

• "Is this output high quality?"
• "Rate the overall effectiveness of this response"
• "How good is this content?"

2. Pass/Fail Definitions

Define exactly what constitutes pass and fail. Always binary — no Likert scales, no 1-5 ratings, no "partially meets criteria."

PASS: The blog post uses formal language throughout. Professional vocabulary,
complete sentences, no slang, no contractions, no first-person casual asides.