Evidence Standards — Appraisal & Balance (amann-evidence-standards)

Evidence Standards — Appraisal & Balance (amann-evidence-standards)

When to trigger

• The framework is set and you are filling cells with the literature
• The field has rival schools, a live controversy, or conflicting empirical results
• You must decide whether to cite everything or curate, and how to weigh disagreeing studies
• You are a contributor to this literature and worry the review tilts toward your own work

At Annals you are the field's reviewer-of-record

An Annals review carries no estimates of its own — but it is not therefore neutral. The "attitude" is disciplined critical appraisal: you judge how good the cumulative evidence is, reconcile conflicts by credibility, and state where the field's confidence is and is not warranted. This is the review-craft replacement for primary-research robustness checks: you are appraising other people's designs, not defending your own.

Completeness vs. selectivity: the Annals contract

A review must be comprehensive in coverage yet selective in emphasis — long (~50 pages) but not an inventory. Resolve the tension by tiering the corpus:

Comprehensiveness is proven by the citation set (the saturation log from amann-literature-synthesis); selectivity is exercised in the prose. Equal-length summaries of every paper abdicate the editorial judgment that is the review's value.

Weighing conflicting evidence (not vote-counting)

Management findings conflict constantly. Reconcile them by why they differ, never by tally:

• Weigh by credibility. Design strength (causal vs. correlational), sample, measurement validity, replication, and publication-bias risk — not "9 studies positive, 4 negative."
• Separate by construct and context. Studies often disagree because they measure different constructs or run in different settings; surface that rather than forcing a false consensus.
• Flag the cumulative-evidence quality. State plainly where the evidence is strong, mixed, or thin — and where apparent consensus rests on a few non-independent studies.
• Name what would settle it. For a live debate, specify the study design that would resolve it (this seeds the agenda).

When appraisal needs an actual number (execution bridge)

Occasionally a credibility judgment cannot be settled by reading alone: the review's account of a controversy hinges on whether a staggered-adoption TWFE estimate survives modern corrections, or an apparent consensus may melt once publication bias is priced in. When a load-bearing magnitude comes with a replication package, audit it rather than adjudicate by prose — the shared playbook execution-with-mcp maps each design family to callable StatsPAI / Stata MCP tools (bacon_decomposition to expose bad-comparison weighting, callaway_santanna to re-estimate, honest_did_from_result for pre-trend fragility). Any number produced this way must come from an actual run and be labelled as the review's own re-analysis. This is the exception, not the Annals default: most appraisal here stays qualitative.

Fairness across schools and the self-citation trap