Evidence Appraisal & Balance (arecon-evidence-standards)

Evidence Appraisal & Balance (arecon-evidence-standards)

When to trigger

• The framework is set and you are filling cells with conflicting empirical results
• You must decide whether "the literature finds X" is actually supported, or only loosely
• The field has rival schools, a live controversy, or estimates that disagree
• You are a contributor to this literature and worry the review tilts toward your own work

Appraising evidence you did not produce

An ARE review runs no identification of its own — there is no design to defend, no replication package of your data. Instead you act as the field's referee-of-record for adjacent readers: you judge how much weight each primary study can bear so the review weighs the evidence correctly. Make the appraisal explicit, not implicit:

• DID / event study: does the cited paper use TWFE on staggered timing without addressing heterogeneity bias? A study that predates Callaway–Sant'Anna / Sun–Abraham corrections may not support the magnitude you attribute to it — flag it.
• IV: is the first stage strong and the exclusion restriction defended, or is it a convenience instrument? Weak-IV results carry less weight in your synthesis.
• RDD: density/manipulation checks, bandwidth robustness — a fragile RDD is a fragile data point.
• Structural / calibration: are the parameters identified and the counterfactual policy-invariant, or is it calibration presented as estimation?
• Experiments: pre-registration, balance, attrition, multiple-hypothesis adjustment, external validity.

You are not re-running these — you are rating their credibility in the evidence matrix so the review's conclusions track the best evidence, not the loudest paper.

Execution bridge — when rating is not enough

A few moments in an ARE review justify running something instead of only rating it: a survey table pools estimates and wants a formal meta-analytic average or publication-bias check; a pivotal magnitude predates the staggered-DiD corrections and its replication data are public, so you can report what callaway_santanna or bacon_decomposition actually does to it rather than speculate; or a weak-instrument worry could be settled by an effective_f_test on the archived first stage. For these, hand off to execution-with-mcp, which maps each design and reviewer objection to the callable StatsPAI / Stata MCP chain (detect_design → fit with as_handle=true → audit_result). Label any such figure as your re-analysis, and never report a number you did not compute.

Weighing, not vote-counting

Conflicting results are reconciled by credibility and by what each study estimates, never by tallying "7 studies positive, 4 negative." Two estimates that disagree often measure different objects (different populations, estimands, time horizons); say so, and let the framework's cells carry the distinction. A pooled "consensus" across non-comparable designs manufactures false agreement that ARE's methodologically literate readers will catch.

Comprehensiveness vs. selectivity: the ARE contract

A review must be comprehensive in coverage yet selective in emphasis — and stay accessible in ~25–40 pages. Tier the corpus: