---
name: evidence-grading
description: "Ranks each source by evidence strength before relying on it — primary vs secondary, peer-reviewed vs preprint vs blog, study design and sample size, recency and conflict of interest — and flags weak evidence instead of treating every source as equally authoritative. Use when someone hands over a mix of studies, articles, posts, or links and asks what the evidence actually shows or how much to trust each. Do NOT use for answering strictly from a given context and citing it (that is rag-grounding-and-citation), for formatting citations, or for summarizing a single source."
metadata:
  decimalai-skill-id: "554c3196-3a24-41c1-8185-511c7ce3bb88"
  decimalai-version-id: "36694aa0-7c6d-4090-aa14-2c1ad23ba9d4"
  decimalai-version: "1"
  decimalai-body-sha256: "aa9eb7929f0cdab057cd0268d53b4afbd30fd6526ebb90171ebed693d95b77e1"
  decimalai-name: "evidence-grading"
  decimalai-format: "1"
  decimalai-invocation: "model"
  decimalai-category: "research"
  decimalai-source: "https://app.decimal.ai/s/evidence-grading@1/SKILL.md"
---
# Grade the Evidence Before You Rely on It

Handed several sources on a question, the base model tends to blend them into one fluent answer as if a personal blog, a preprint, and a peer-reviewed meta-analysis all carried the same weight — and to over-trust whichever source is most recent or most confidently worded. This skill enforces the appraisal step that librarians, clinicians, and systematic reviewers do by habit: place each source on a recognized evidence hierarchy, weight the conclusion by the strongest support, and flag weak evidence out loud *before* leaning on it.

## When to Activate

Activate when the user presents information of mixed or unstated quality and asks how much to trust it or what the evidence really shows:
- Several sources that disagree — "these say opposite things, which do I believe?"
- A pile of links/studies/posts on a topic and "what does the research actually say?"
- One striking or recent claim they are tempted to act on
- A claim backed by a source with an obvious stake in it (a vendor, a seller)

Do NOT activate to summarize a single source, to format references, or to answer only from a provided context (that is grounding, not appraisal).

## The Evidence Hierarchy

Rank source **type**, strongest to weakest. This ordering is the default; a specific source can move up or down on the appraisal dimensions below.

1. **Systematic review / meta-analysis** of multiple studies, peer-reviewed — the strongest single source.
2. **Peer-reviewed primary study**, ranked by design: randomized controlled trial > prospective cohort > case-control > cross-sectional/observational.
3. **Peer-reviewed but weak design** — small sample, no control group, case series, single case report.
4. **Preprint / conference abstract** — not yet peer-reviewed; findings are provisional.
5. **Official reports** (government agency, standards body) and **reputable journalism that cites primaries** — reliable but usually secondary.
6. **Textbooks and review articles** — secondary, authoritative but may lag current findings.
7. **Credentialed expert opinion / blog** — a named expert reasoning, but unreviewed.
8. **Marketing/vendor content, anonymous blogs, forum posts, social media, single anecdotes** — weakest; assume potential conflict of interest.

## Appraise Each Source on These Dimensions

Type is the starting point; adjust with:

- **Primary vs secondary** — is this original data/analysis, or a retelling of someone else's? Trace a big secondary claim back to the primary before trusting it.
- **Peer-reviewed vs not** — a preprint or blog result is provisional no matter how striking.
- **Study design & sample** — controlled beats uncontrolled; larger, relevant samples beat tiny or off-population ones. A large *observational* study still cannot show causation the way a small RCT can — do not rank on sample size alone.
- **Recency** — current enough for a fast-moving field, but newer is not stronger: a fresh preprint does not outrank an established meta-analysis.
- **Independence / conflict of interest** — who funded and published it, and do they profit from the conclusion?
- **Convergence** — multiple *independent* strong sources agreeing is worth more than any one of them alone.

## The Discipline (what the base skips by default)

1. **Label each source's tier before you use it.** Do not treat a blog and a trial as equal inputs.
2. **Weight the conclusion by the strongest support**, and say plainly when a claim rests only on weak sources.
3. **Flag weak evidence explicitly before relying on it**, and name what would raise confidence (peer review, replication, a controlled trial).
4. **Never let recency or a confident tone stand in for strength.**
5. **Prefer primary sources**; when a claim matters, trace it to the primary rather than citing the summary.
6. **When strong sources genuinely conflict, say so** — do not average them into a false consensus.

## Output Shape

When weighing a set of sources, structure the answer as:

```markdown
**Sources, graded:**
- [source] — [tier: e.g. peer-reviewed RCT / preprint / vendor blog] — [strong | moderate | weak], because [design/sample/independence]
**What the strong evidence supports:** [conclusion carried by the top sources]
**Weak or provisional:** [which claims rest only on weak sources, and what would upgrade them]
**Verdict:** [answer, with confidence set by the strongest support — not the loudest source]
```

## Example

### Input
```text
Does vitamin D cut the risk of catching a cold? I found:
[1] a 2017 meta-analysis of 25 trials in a peer-reviewed journal — small overall benefit
[2] a wellness influencer's post saying it "basically stops you getting sick"
[3] a 2024 preprint, 18 people, claiming a huge effect
Which should I go with?
```

### Expected Response

**Sources, graded:**
- [1] Meta-analysis of 25 randomized trials, peer-reviewed — **strong**; pools many controlled studies, the top of the hierarchy.
- [2] Influencer social post — **weak**; unsourced, confident tone is not evidence, and there may be a stake in selling supplements.
- [3] Preprint, n=18 — **weak/provisional**; not peer-reviewed and far too small to trust a "huge effect."

**What the strong evidence supports:** [1] is the anchor — a modest reduction in risk, not dramatic protection.

**Weak or provisional:** the influencer's claim and the tiny preprint both point to a large effect, but neither can carry that weight. The preprint would firm up only after peer review and replication in a larger sample.

**Verdict:** the honest answer is [1]'s — a small benefit — even though [2] and [3] are more exciting. Do not let the newer, louder sources override the reviewed, pooled one.

## Edge Cases

- **Everything is weak.** If the only sources are blogs and anecdotes, say the question is currently unsupported by strong evidence rather than manufacturing a confident answer from them.
- **Two strong sources disagree.** Two peer-reviewed meta-analyses can genuinely conflict. Report the conflict and look at scope, population, or method differences — don't pick one arbitrarily or split the difference.
- **A large observational study vs a small trial.** Size is not design: the observational study may be confounded; note what each can and cannot establish instead of ranking by n.
- **The news vs the paper.** A headline often overstates the study under it. Grade and quote the primary study's actual claim, not the reporting.
- **One promising study.** A single small study is preliminary; flag that acting on it needs replication or convergence, however plausible it reads.

## Evaluation Criteria
A good execution of this skill should:
- [ ] Assign each source a strength tier before drawing a conclusion, not treat them as equal
- [ ] Weight the verdict by the strongest sources rather than the most recent or most confident
- [ ] Flag the weak/provisional sources explicitly and say what would raise confidence
- [ ] Distinguish primary from secondary, peer-reviewed from preprint, controlled from observational
- [ ] Report genuine conflicts between strong sources instead of forcing a consensus