Write Labeling Guidelines That Produce Consistent Labels

Asked to write instructions for labeling a dataset, the base model tends to produce a thin rubric: a list of label names with a one-line gloss each, no boundaries, no examples of the hard cases, and no rule for what an annotator does when stuck. Handed to two people, that rubric produces two different datasets — the disagreement lands in the middle, exactly where the one-liners run out. The label noise this creates propagates into whatever model trains on the data.

A usable guideline is the discipline that professional annotation teams follow: definitions drawn at the boundary, the hard cases worked out in advance, a decision rule for ties, canonical examples, and a way to measure whether annotators actually agree. This skill supplies that structure.

What a Usable Guideline Contains

Every labeling guideline needs all six parts below. The base supplies the first (label names) by default and skips the rest — those are where consistency is won.

1. Define each label by its boundary, not just its name

A name ("negative", "urgent", "spam") is not a definition. State what falls inside the label and what falls outside it — especially against the adjacent label it is most often confused with. The boundary between two labels is where annotators disagree, so that is what the definition must pin down.

> Labeling a piece of writing as opinion / reporting / mixed: > - reporting — states verifiable facts and attributes them to sources; contains no evaluative language from the author. Excludes pieces that draw a conclusion the sources did not state. > - opinion — the author argues for a position or makes value judgments. Excludes pieces that merely quote someone else's opinion while staying neutral. > - mixed — both a factual core and the author's own argument are present and neither is incidental.

Aim for labels that are mutually exclusive (a case fits one) and exhaustive (every case fits some label — add an other / none label if not).

2. Give a worked edge case for each label

For every label, include at least one borderline example — a case that looks like it could be something else — and show which label it takes and why. Edge cases are the actual content of a guideline; the obvious cases never needed instructions.

> - "Officials said the policy will save money." → reporting (attributed factual claim, no authorial judgment), not opinion, even though it sounds favorable. > - "This reckless policy will bankrupt us." → opinion (value words "reckless", "bankrupt" are the author's), not reporting.

3. Add a tie-break rule for when two labels fit or nothing fits

Annotators freeze on the ambiguous middle. Remove the freeze with an explicit rule. Two standard forms:

• Priority order — "if a case satisfies both X and Y, label it X." (Use when one label should dominate.)
• Default label — "if you cannot decide after re-reading, label it other / unsure." (Use so genuinely ambiguous cases collect in one predictable place instead of being split randomly.)

State plainly what to do when the case matches no label, too. Never leave "what if I'm not sure" unanswered — that silence is where consistency dies.

4. Include gold-standard exemplars

For each label, give one or two canonical, unambiguous examples — the clearest possible instance. New annotators calibrate against these, and they double as the answer key for onboarding. Keep exemplars separate from the edge cases: exemplars show the center of a label, edge cases show its border.

5. Specify the label set and format up front

Name the exact label set (and whether an item can take more than one label). If labels are ordered (e.g. severity 1–5), say what each level means at the anchor points, not just the endpoints. Ambiguity about which labels exist is a silent source of disagreement.

6. Add an inter-annotator agreement check

A guideline is unverified until two or more annotators independently label the same sample and you measure how often they match. This is the step the base always omits.

• Have ≥2 annotators label a shared set (e.g. 50–100 items) blind to each other.
• Report percent agreement and, better, Cohen's kappa (or Fleiss' kappa for >2 annotators), which corrects for agreement expected by chance. A common working bar is kappa ≥ 0.7; below ~0.4 the labels are barely better than chance.
• Low agreement means the guideline is ambiguous, not that the annotators are careless. Find the label pairs they confuse, sharpen those boundaries and edge cases, and re-measure. Iterate the guideline, not the people.

Output Shape

Deliver the guideline as a document an annotator can work straight from:

markdown
## Task
[one sentence: what is being labeled and the exact label set]

## Labels
### <label> — <one-line definition>
Include: … / Exclude: …  (boundary vs the nearest other label)
Gold example: …
Edge case: <borderline item> → this label, because …
[repeat per label]

## When unsure
[tie-break / priority order / default label; what to do if nothing fits]

## Agreement check
[how many items double-labeled, by whom, which metric, the target, what to do if it's low]

Edge Cases

• A label almost never occurs. Rare labels get inconsistent treatment from lack of practice. Give them extra gold examples and flag them for a second review rather than dropping them.
• Order effects / fatigue. If items are labeled in sequence, note that annotators drift; recommend shuffling and periodic re-calibration against the gold set.
• Subjective tasks (sentiment, toxicity, offensiveness). Perfect agreement is not the target; define the boundary as tightly as the task allows, then report the residual disagreement honestly instead of pretending a single truth exists.
• Evolving label set. If annotators keep hitting cases that fit nothing, that is a signal to add a label — but re-label the earlier batch under the new set, don't leave the dataset split across two schemas.
• One annotator. With a single labeler you cannot measure agreement at all; have them re-label a sample after a delay (intra-annotator consistency) as a weak substitute and say so.

Evaluation Criteria

A good execution of this skill should:

• ] Define each label by an explicit in/out boundary against its nearest neighbor, not a bare name
• ] Supply at least one worked borderline example per label
• ] State a tie-break rule for cases that fit two labels and a default for cases that fit none
• ] Provide canonical gold exemplars distinct from the edge cases
• ] Include a concrete inter-annotator agreement check and say to fix the guideline when agreement is low