Evidence Grading Skill

"We have evidence" covers everything from a randomized experiment to the CEO's seatmate on a flight — and decisions made on ungraded evidence inherit the confusion. The grading discipline: inventory what actually supports the claim, place each item on the business-evidence hierarchy (what it is, not how confident it feels), test the graded weight against the decision's stakes (a reversible pilot needs less than a one-way rebrand — decision-journal logic), and when evidence falls short, name the cheapest upgrade — because the answer to weak evidence is usually a better test, not a braver bet.

What This Skill Produces

• The evidence inventory — everything supporting (and contradicting) the claim, each item graded on the hierarchy
• The weight assessment — what the graded pile actually supports, including the mixed-signals honest read
• The sufficiency verdict — enough for this decision's stakes / not yet — with the stakes analysis shown
• The upgrade path — the cheapest next evidence that would move the verdict ("a 2-week holdout test settles this for $0")

Required Inputs

Ask for these if not provided:

• The claim and the decision riding on it — "users want X" feeding a backlog item vs. feeding a repositioning are different sufficiency bars; the decision's reversibility and cost set the bar
• The evidence, itemized — every piece: the data pull, the survey, the five customer quotes, the competitor's move, the expert's opinion — including the inconvenient items (an inventory that omits contradicting evidence is advocacy)
• The evidence's provenance — n, selection, dates, who collected it and with what incentive (source-triangulation supplies the externals; internal evidence has incentives too)

Framework: The Grading Rules

1. The hierarchy, applied without sentiment: experiments/A-B tests (causal) > usage/behavioral data (what people do, correlational) > surveys (what they say, at scale — survey-design-basics quality adjusts the grade) > interviews (rich, small-n — interview-synthesis counts matter) > anecdotes (existence proofs only) > expert opinion (informed priors) > internal conviction (a hypothesis, not evidence). Each item gets its rung and its quality-within-rung — a leading survey grades below honest interviews.
2. Direction and independence both count: the inventory includes contradicting evidence at its own grade, and echo-checks the supporting pile (three anecdotes traceable to one loud customer are one anecdote). The weight is the net, honestly netted.
3. Anecdotes prove existence, never prevalence: "a customer asked for X" establishes that the need exists somewhere — it cannot establish how common; the classic grading error is prevalence conclusions from existence evidence, and it's the error this skill most often catches.
4. Sufficiency is relative to stakes: the verdict tests weight against the decision — reversible-and-cheap decisions legitimately run on interview-grade evidence (the pilot is the experiment); irreversible-and-expensive ones demand behavioral or experimental grade. "Weak evidence" isn't a verdict; "weak for this bet" is.
5. The upgrade path is the constructive ending: when insufficient, name the cheapest test that would change the verdict — the holdout, the fake-door, the survey that sizes the interview theme, the pilot-with-metrics. Ranked by cost-to-confidence ratio; the skill's product is often not "no" but "this $0 two-week test first."

Output Format

Evidence Grade: "the claim]" — feeding the decision]

The Inventory

| Evidence | Rung | Quality notes (n, selection, date, independence) | Direction | |---|---|---|---|

The Weight

What the graded net actually supports, in one honest paragraph — existence vs. prevalence vs. causation explicitly]

The Sufficiency Verdict

The decision's stakes (reversibility × cost) · enough / not yet · the reasoning]

The Upgrade Path

The cheapest evidence that moves the verdict · cost and time · the second option]

Quality Checks

• ] Every item has a rung and within-rung quality notes
• ] Contradicting evidence is in the inventory at its own grade
• ] Echoes were collapsed before weighing
• ] The verdict is stakes-relative, not absolute
• ] The insufficient branch ends in a priced upgrade, not just a no

Anti-Patterns

• ] Do not grade by vividness — the memorable anecdote outshines the boring dataset in every meeting; the hierarchy exists to resist exactly that
• ] Do not conclude prevalence from existence — the most common grading felony
• ] Do not omit the inconvenient items — an advocacy inventory grades the author, not the claim
• ] Do not demand experimental grade for reversible bets — over-evidencing cheap decisions is its own waste
• ] Do not end at "insufficient" — the upgrade path is the difference between rigor and obstruction