---
name: mohitagw15856/eval-rubric-designer
source: https://app.decimal.ai/s/mohitagw15856-eval-rubric-designer@1/SKILL.md
source_sha256: bd961e8647de
---

# Eval Rubric Designer Skill

You can't improve what you can't score. The hard part of evaluating AI output isn't running the judge — it's
defining dimensions that are **specific, observable, and independent**, with anchors concrete enough that two
people (or two judge runs) agree. This skill turns "is the output good?" into a rubric and a judge prompt you
can run today.

## Working from a brief

Given just "I need to eval my summariser", **produce the full rubric anyway** — infer the task, the output
type, and the dimensions that matter for it, and label inferred choices. Never hand back a list of dimension
names with no anchors; the anchors are where the rubric earns its keep.

## Required Inputs

Ask for these only if they aren't already provided (else infer and label):

- **The task** — what the AI is supposed to produce, and for whom.
- **A sample output (or two)** — ideally one good and one weak, to calibrate anchors.
- **What "good" means here** — the quality bar and any non-negotiables (e.g. must be grounded, must follow format).
- **How it'll be scored** — human review, LLM-as-judge, or both; and whether you need a single score or per-dimension.

## Output Format

### Eval Rubric: [task]

**1. Dimensions** — 3–6 independent dimensions, each with a one-line definition and a weight. Default set,
tailored to the task: **structure**, **completeness**, **correctness/grounding**, **usefulness**, **safety/tone**.

**2. Anchors** — for each dimension, concrete descriptions at **1, 3, and 5** (what a poor / acceptable /
excellent answer looks like *for this task*). Anchors must be observable, not "feels good".

| Dimension (weight) | 1 — poor | 3 — acceptable | 5 — excellent |
|---|---|---|---|
| Grounding (×2) | invents facts not in the source | mostly grounded, minor drift | every claim traceable to the source |

**3. Judge prompt** — a ready-to-run LLM-as-judge prompt in a fenced block: the task description, the rubric,
an instruction to score each dimension 1–5, and a **strict JSON output contract** (`{"dimension":N,...}`) so
scores parse reliably. Include a one-line "return only JSON" reinforcement.

**4. Labelling guide** — short rules for tie-breaks and common edge cases, so repeat runs stay consistent.

**5. Judge reliability notes** — known biases (length, position, self-preference), and how to mitigate: a
cheaper judge for scale vs. a stronger judge for the rubric, sampling N runs, and spot-checking judge scores
against a few human labels before trusting the leaderboard.

## Quality Checks

- [ ] Dimensions are independent — a single flaw doesn't tank three of them at once
- [ ] Every dimension has concrete 1/3/5 anchors specific to this task, not generic adjectives
- [ ] The judge prompt has a strict, parseable output contract (JSON), with a retry/repair note
- [ ] Weights reflect what actually matters for the task (grounding usually > prose polish)
- [ ] The rubric is calibrated against at least one good and one weak sample
- [ ] Judge biases are named with a concrete mitigation, not just listed

## Anti-Patterns

- [ ] Do not ship dimension names without anchors — names alone don't make scores reproducible
- [ ] Do not let one quality issue load onto multiple dimensions — keep them orthogonal
- [ ] Do not trust an LLM judge blind — calibrate against a handful of human labels first
- [ ] Do not use a vague "overall quality 1–10" — it hides which part is broken
- [ ] Do not ignore the negative case — a rubric must distinguish "wrong" from "thin", not just "great" from "okay"

## Based On

LLM-as-judge evaluation practice — orthogonal weighted dimensions, anchored scales, structured judge prompts, and judge-bias mitigation.