Robustness Test Checklist

Systematic checklist of robustness tests for empirical research. Use this to ensure comprehensive testing before submission.

Core Robustness Categories

1. Alternative Samples

| Test | Description | When to Use | |------|-------------|-------------| | Exclude outliers | Winsorize/trim at different levels (0.5%, 2%, 5%) | Always | | Drop financial firms | Exclude SIC 6000-6999 | If not already excluded | | Drop regulated industries | Exclude utilities, telecoms | Industry-specific effects | | Different time periods | Split sample pre/post crisis, early/late | Results may be period-specific | | Geographic subsamples | By region, state, country | External validity | | Size subsamples | Small vs. large firms | Heterogeneous effects | | Balanced panel | Require continuous observations | Survivorship concerns |

2. Alternative Specifications

| Test | Description | When to Use | |------|-------------|-------------| | Different fixed effects | Firm, industry×year, state×year | Control for unobservables | | Additional controls | Add variables referees might suggest | Omitted variable concerns | | Drop controls | Verify not over-controlling | Mediator concerns | | Different clustering | Firm, industry, state, two-way | Inference robustness | | Different standard errors | Bootstrap, Newey-West, Driscoll-Kraay | Serial/cross-sectional correlation | | Nonlinear specifications | Quadratic terms, splines | Linearity assumption | | Log vs. level | Transform dependent variable | Skewed distributions |

3. Alternative Measures

| Test | Description | When to Use | |------|-------------|-------------| | Alternative dependent variable | Different proxy for same concept | Measurement concerns | | Alternative treatment measure | Continuous vs. binary, different threshold | Treatment definition | | Alternative control measures | Different proxies for size, leverage, etc. | Standard practice | | Scaled differently | By assets, sales, employees | Scaling choice matters |

4. Identification Tests

| Test | Description | When to Use | |------|-------------|-------------| | Placebo/Falsification | | | | Placebo timing | Fake treatment 1-3 years before actual | DiD parallel trends | | Placebo outcome | Effect on outcome that shouldn't be affected | Specificity of mechanism | | Placebo treatment | Random assignment of treatment | Rule out spurious correlation | | Pre-trends | | | | Event study plot | Coefficient for each pre/post period | Visual parallel trends | | Joint F-test | Test pre-period coefficients = 0 | Statistical parallel trends | | Endogeneity | | | | Instrumental variables | Find exogenous variation | Selection concerns | | Heckman selection | Model selection explicitly | Sample selection | | Propensity score matching | Match treated/control | Observable selection | | Entropy balancing | Reweight to balance covariates | Covariate imbalance | | Regression discontinuity | If threshold exists | Sharp identification |

5. Inference Robustness

| Test | Description | When to Use | |------|-------------|-------------| | Wild cluster bootstrap | Small number of clusters | <50 clusters | | Randomization inference | Permutation-based p-values | Few treated units | | Conley standard errors | Spatial correlation | Geographic data | | Multiple hypothesis correction | Bonferroni, FDR | Many outcomes tested |

DiD-Specific Tests

For difference-in-differences designs:

• ] Event study with pre-treatment coefficients
• ] Parallel trends test (formal)
• ] Bacon decomposition (staggered treatment)
• ] Callaway-Sant'Anna or Sun-Abraham estimator (heterogeneous treatment effects)
• ] Placebo treatment timing
• ] Vary treatment window
• ] Triple-difference if possible
• ] Exclude always-treated or never-treated

IV-Specific Tests

For instrumental variables:

• ] First-stage F-statistic (>10, prefer >100)
• ] Weak instrument robust inference (Anderson-Rubin)
• ] Overidentification test (if multiple IVs)
• ] Exclusion restriction discussion
• ] Reduced form results
• ] Compare OLS vs. IV magnitudes

Quick Commands

• "robustness checklist" - Full checklist for current paper
• "DiD robustness" - DiD-specific tests only
• "what tests for method]" - Tests for specific identification strategy
• "referee-proof" - Most commonly requested tests
• "prioritize tests" - Rank by importance for your setting

Referee-Proof Minimum

At minimum, most papers should include:

1. Sample robustness: Exclude outliers, alternative time periods
2. Specification robustness: Alternative fixed effects, controls
3. Measurement robustness: Alternative variable definitions
4. Inference robustness: Alternative clustering
5. Identification test: At least one placebo or pre-trend test

Reporting Template

For robustness tables:

Table X: Robustness Tests
Panel A: Alternative Samples
  (1) Baseline
  (2) Exclude financial firms
  (3) Exclude 2008-2009
  (4) Winsorize at 5%

Panel B: Alternative Specifications
  (5) Add industry×year FE
  (6) Control for firm age
  (7) Cluster by industry

Panel C: Alternative Measures
  (8) Alternative dependent variable
  (9) Continuous treatment measure