--- name: dataset-labeling-guidelines source: https://app.decimal.ai/s/dataset-labeling-guidelines@1/SKILL.md source_sha256: 9382e85ac8f6 --- # Write Labeling Guidelines That Produce Consistent Labels Asked to write instructions for labeling a dataset, the base model tends to produce a thin rubric: a list of label names with a one-line gloss each, no boundaries, no examples of the hard cases, and no rule for what an annotator does when stuck. Handed to two people, that rubric produces two different datasets — the disagreement lands in the middle, exactly where the one-liners run out. The label noise this creates propagates into whatever model trains on the data. A usable guideline is the discipline that professional annotation teams follow: definitions drawn at the boundary, the hard cases worked out in advance, a decision rule for ties, canonical examples, and a way to *measure* whether annotators actually agree. This skill supplies that structure. ## What a Usable Guideline Contains Every labeling guideline needs all six parts below. The base supplies the first (label names) by default and skips the rest — those are where consistency is won. ### 1. Define each label by its boundary, not just its name A name ("negative", "urgent", "spam") is not a definition. State what falls **inside** the label and what falls **outside** it — especially against the *adjacent* label it is most often confused with. The boundary between two labels is where annotators disagree, so that is what the definition must pin down. > Labeling a piece of writing as `opinion` / `reporting` / `mixed`: > - **reporting** — states verifiable facts and attributes them to sources; contains no evaluative language from the author. *Excludes* pieces that draw a conclusion the sources did not state. > - **opinion** — the author argues for a position or makes value judgments. *Excludes* pieces that merely quote someone else's opinion while staying neutral. > - **mixed** — both a factual core and the author's own argument are present and neither is incidental. Aim for labels that are **mutually exclusive** (a case fits one) and **exhaustive** (every case fits some label — add an `other` / `none` label if not). ### 2. Give a worked edge case for each label For every label, include at least one **borderline example** — a case that looks like it could be something else — and show which label it takes and why. Edge cases are the actual content of a guideline; the obvious cases never needed instructions. > - *"Officials said the policy will save money."* → **reporting** (attributed factual claim, no authorial judgment), not opinion, even though it sounds favorable. > - *"This reckless policy will bankrupt us."* → **opinion** (value words "reckless", "bankrupt" are the author's), not reporting. ### 3. Add a tie-break rule for when two labels fit or nothing fits Annotators freeze on the ambiguous middle. Remove the freeze with an explicit rule. Two standard forms: - **Priority order** — "if a case satisfies both X and Y, label it X." (Use when one label should dominate.) - **Default label** — "if you cannot decide after re-reading, label it `other` / `unsure`." (Use so genuinely ambiguous cases collect in one predictable place instead of being split randomly.) State plainly what to do when the case matches *no* label, too. Never leave "what if I'm not sure" unanswered — that silence is where consistency dies. ### 4. Include gold-standard exemplars For each label, give one or two **canonical, unambiguous** examples — the clearest possible instance. New annotators calibrate against these, and they double as the answer key for onboarding. Keep exemplars separate from the edge cases: exemplars show the center of a label, edge cases show its border. ### 5. Specify the label set and format up front Name the exact label set (and whether an item can take more than one label). If labels are ordered (e.g. severity 1–5), say what each level means at the anchor points, not just the endpoints. Ambiguity about *which labels exist* is a silent source of disagreement. ### 6. Add an inter-annotator agreement check A guideline is unverified until two or more annotators independently label the **same** sample and you measure how often they match. This is the step the base always omits. - Have ≥2 annotators label a shared set (e.g. 50–100 items) blind to each other. - Report **percent agreement** and, better, **Cohen's kappa** (or Fleiss' kappa for >2 annotators), which corrects for agreement expected by chance. A common working bar is kappa ≥ 0.7; below ~0.4 the labels are barely better than chance. - **Low agreement means the guideline is ambiguous, not that the annotators are careless.** Find the label pairs they confuse, sharpen those boundaries and edge cases, and re-measure. Iterate the *guideline*, not the people. ## Output Shape Deliver the guideline as a document an annotator can work straight from: ```markdown ## Task [one sentence: what is being labeled and the exact label set] ## Labels ###