--- name: brycewang-stanford/aer-preregistration source: https://app.decimal.ai/s/brycewang-stanford-aer-preregistration@1/SKILL.md source_sha256: 98ef29fa1611 --- # AER Pre-Registration ## Overview For experimental and prospective-data AER-track work, credibility is bought **before** the data exist. This skill writes the pre-analysis plan (PAP), sizes the sample from a power calculation, and registers the study. Its job is to make the eventual results un-p-hackable: a referee who sees a public PAP timestamp predating data collection cannot accuse you of specification search. Registration is not optional at AEA journals for RCTs. If the design is observational, skip to `aer-robustness`; there is nothing to pre-register. ## When to Use - The paper runs a randomized controlled trial (field, lab, or survey experiment) - The project collects primary data whose analysis should be pre-committed - A referee or editor asks for the trial registration number or the PAP - Reviewers suspect the reported specification was chosen after seeing outcomes - You must justify the sample size, or explain why an effect was "not detected" ## Register-or-Not Decision ``` Is treatment assigned by the researcher (randomization)? ├── Yes → register with the AEA RCT Registry BEFORE the intervention; write a PAP ├── No, but you collect new primary data → post a PAP for the pre-committed analysis └── No, secondary/observational data already realized → do NOT pre-register (a PAP written after the outcomes exist is theater); go to aer-robustness ``` ## Pre-Analysis Plan Contents Pre-specify, in order, and timestamp before unblinding: 1. **Hypotheses** — each stated as a signed, testable prediction, not a topic: > H1: the cash transfer raises household consumption at endline by at > least 0.15 SD, estimated by OLS of log consumption on treatment with > strata fixed effects, clustered at the village level. 2. **Primary outcomes** — a short list (ideally 1-3). Everything else is secondary or exploratory and labeled as such. 3. **Estimating equation** — the exact regression, fixed effects, covariates, and the level at which standard errors cluster. 4. **Sample and inclusion rules** — who is in the analysis and how attrition is handled (pre-commit to bounds; see `examples/lee-bounds-demo/`). 5. **Multiple-testing correction** — the family and the method (e.g. Romano-Wolf, `romano_wolf_2005`), pre-specified, not chosen after the p-values land. 6. **Heterogeneity** — the subgroups you will test, fixed in advance; all others are exploratory. 7. **Power** — the MDE and the assumptions behind it (below). Keep the PAP moderate in scope (Olken's advice): pre-specify the primary analysis tightly, leave genuine discovery clearly flagged as exploratory. An over-long PAP that pre-registers forty outcomes protects nothing. ## Power and the Minimum Detectable Effect Size the sample from the MDE, not the other way around. For a two-arm trial with equal allocation, size `sigma`, share `p`, total `N`: ``` MDE = (z_power + z_{1-alpha/2}) * sigma * sqrt(1 / (p (1 - p) N)) ``` - Target **80% power** at a **5%** two-sided level: $z_{0.80} = 0.84$ and $z_{0.975} = 1.96$, so `z_power + z_alpha = 2.80`. - State the MDE in the units the audience cares about, and benchmark it against the smallest effect that would be economically interesting. If the MDE exceeds that, the study is underpowered — do not run it as designed. - Correct the variance for **clustered** assignment (design effect $1 + (m-1)\rho$ — e.g. $m = 30$ per cluster and $\rho = 0.05$ inflates the needed sample by a factor of $2.45$), for a baseline covariate ($R^2$ gain), and for expected attrition. A power number that ignores the ICC is fiction. - Underpowered designs do not just miss: when they reach significance they **exaggerate** the effect (Type-M). See `examples/power-mde-demo/` for the MDE-attains-target-power check and the winner's-curse simulation. Cite `mckenzie_2012` (more rounds beat larger cross-sections when outcomes are noisy) and `duflo_glennerster_kremer_2007` (the design toolkit). Keys in [`../../references.bib`](../../references.bib); defaults in [`../../docs/methods-reference.md`](../../docs/methods-reference.md). ## Reporting Against the Plan - Report the pre-specified primary result **first**, exactly as written, even if it is null. A pre-registered null is a publishable finding, not a failure. - Mark every deviation from the PAP explicitly, with the reason, in a table. - Separate confirmatory from exploratory results in the manuscript; never promote an exploratory subgroup to the headline. - File the disclosure and IRB/ethics approvals; AEA journals require both. ## Red Flags for Referees - No registration number on an RCT, or a timestamp *after* data collection began - A power calculation that omits clustering, attrition, or the ICC - Twenty "primary" outcomes with no multiplicity correction - The headline result is an unregistered subgroup interaction - "Not statistically significant" reported as "no effect" with no MDE stated - Deviations from the PAP that are silent rather than disclosed ## Pre-Registration Gate Do not advance to data collection until **all** are true: - [ ] Primary outcomes and the estimating equation are pre-specified in writing - [ ] The sample size is justified by an MDE with clustering and attrition built in - [ ] The multiple-testing family and correction are fixed in advance - [ ] The study is registered (AEA RCT Registry) with a pre-intervention timestamp - [ ] Exploratory analyses are labeled as such, not disguised as confirmatory ## Repository Resources Bundled with the installed skill, no repository checkout needed --- read it before the repo resources below: - `references/pap-template.md` --- PAP outline, power/MDE reporting template, registry field checklist When working from the repo or plugin bundle, load only the relevant resource: - Power/MDE worked simulation with the Type-M winner's-curse check: `examples/power-mde-demo/` - Attrition bounds to pre-commit in the PAP: `examples/lee-bounds-demo/` - Estimator defaults, diagnostics, and BibTeX keys: `docs/methods-reference.md` - Multiple-testing correction methods: `skills/aer-robustness/SKILL.md` - Verified references (`mckenzie_2012`, `duflo_glennerster_kremer_2007`): `references.bib` Fix the MDE and the primary-outcome list before drafting; both feed the `aer-identification` estimator choice and the `aer-consistency` audit. ## Handoff ```text DESIGN: PRIMARY OUTCOMES: MDE / POWER: MULTIPLICITY: REGISTRATION: NEXT SKILL: aer-identification (confirm estimator) then aer-robustness ``` ## Anti-Patterns - Writing the PAP after the endline data are in hand — the timestamp is the point - Pre-registering so many outcomes that "confirmatory" loses all meaning - Powering the study for the effect you hope for instead of the smallest one worth detecting - Treating a registered null as a failed experiment rather than a clean result - Reporting an exploratory subgroup as if it had been pre-specified