Transparent Methods Reporter

Transparent Methods Reporter

Instructions

1. Design Documentation and Pre-Registration

This section covers upstream documentation that makes transparent reporting possible. A well-documented bad design is still a bad design; transparency tools are necessary but not sufficient (Druckman 2022).

• Design Document: Before data collection, create a comprehensive document recording all decisions and their rationale -- motivation, stimuli, outcome measures, predictions, analysis plans, and logistics. This "design document" is the upstream practice that enables downstream transparent reporting (Druckman 2022, Ch. 5).
• Pre-Registration vs. Pre-Analysis Plan: Distinguish between study registration (basic: recording the study's existence, hypotheses, and population in a public repository) and a pre-analysis plan (detailed: specifying exact statistical tests, evidence thresholds, and contingency plans). Require the latter for confirmatory experiments. Name the repository used (OSF, EGAP, or AsPredicted) and provide the registration ID. For PAP structure, cross-reference the pre-registration-writing skill; for estimand, SESOI, and primary/secondary/exploratory classification, cross-reference hypothesis-building.
• Six Core Pre-Registration Elements (Lakens's JARS-aligned summary): For preregistered experiments, ensure the following six elements are specified, following Lakens's (2025) summary of the APA Journal Article Reporting Standards for quantitative research (JARS-Quant; Appelbaum et al. 2018): (1) randomization procedure, (2) inclusion/exclusion criteria, (3) sampling procedures and expected participation rate, (4) sample size justification with power analysis or precision rationale, (5) data diagnostics (exclusion criteria, missing data handling, outlier definitions, assumption checks), and (6) analytic strategy organized into primary, secondary, and exploratory tiers. The primary JARS-Quant tables (Appelbaum et al. 2018, American Psychologist; incorporated into APA 2020, Publication Manual, 7th ed., ch. 3) contain a longer branching item set covering observational, clinical-trial, longitudinal, replication, and N-of-1 designs; the six elements above are Lakens's pedagogical condensation, not JARS-Quant in full. For the broader reproducibility context in which JARS sits, see Munafò et al. (2017, "A Manifesto for Reproducible Science").
• Minimal vs. Complete Pre-Registration. Waldron and Allen (2022) show that minimal pre-registrations (hypotheses only, or under-specified analysis plans) re-admit exploratory researcher degrees of freedom under a confirmatory banner. A pre-registration that lists hypotheses but leaves variables, exclusion rules, or model specifications open is closer to a public hypothesis than a confirmatory PAP. Require complete specification (items 1--6 above) when a study will be reported as confirmatory.
• Analysis Code: The gold standard is to preregister analysis code that runs on a simulated dataset, eliminating ambiguity about all analytical decisions (Lakens 2025). For conjoint analyses, provide the regression specification in code form (e.g., R lm() or feols() call).
• Pilot Documentation: Document all pilot studies, including not just manipulation check results but also response rate data, recruitment language testing, and any modifications made as a result (Druckman 2022, Ch. 5). For conjoints, pilot whether respondents attend to all attributes, find combinations plausible, and process the display as intended.

2. Subjects, Recruitment, and Setting

• Eligibility: Explicitly state who was eligible to participate and the criteria for subject selection.
• Target Population: State the target population to which inference is intended. This includes the units, contexts, measures, and outcomes (Druckman 2022, Ch. 6). Distinguish between the sampling frame (who could be reached) and the target population (who the theory applies to).
• Timeline: Report the exact dates of the recruitment period and when the experiments were conducted, including any repeated measurements or follow-ups. For "firehouse studies" conducted in response to real-world events, document the lag between the event and data collection (Mutz 2011).
• Provider Details: For survey experiments, identify the survey firm used and describe their recruitment methods if they are not universally known. Note whether the sample is probability-based or nonprobability (quota-matched online panel) and the implications for generalizability.
• Response Rates: Provide the response rate and specify the exact formula used for its calculation.
• Survey Error Pipeline: Report how each of three sequential error sources was addressed: (1) coverage error (does the sampling frame reach the target population?), (2) sampling error (does the sample represent the frame?), and (3) nonresponse error (do respondents differ from non-respondents?) (Stantcheva 2023).
• Benchmark Validation: Compare sample demographics and key attitude measures to those from existing high-quality, representative surveys that serve as benchmarks for the target population (Stantcheva 2023).
• Context: Detail the specific setting (e.g., lab, field, online panel) and relevant geographic or institutional characteristics of the population.
• Incentives: Describe the form and amount of any incentives provided to participants (Gerber et al. 2014).

3. Allocation and Treatment

• Randomization Procedure: State clearly if random assignment was used and describe the specific procedure (e.g., simple randomization, blocking, stratification, or restrictions). Identify the software or tool used for randomization (Gerber et al. 2014).
• Unit of Randomization: Explicitly define the unit of randomization -- whether individuals, households, groups, or clusters.
• Assignment Sequence: Provide details on the exact randomization sequence: who generated it, when it was generated, and whether it was concealed from researchers during enrollment (Gerber et al. 2014).
• Blinding: Report whether single-blinding (subjects unaware of condition), double-blinding (subjects and analysts unaware), or no blinding was used (Gerber et al. 2014).
• Baseline Balance: Provide a table of baseline means and standard deviations for demographic characteristics and other pretreatment measures across all experimental groups to detect potential errors in assignment.
• Intervention Detail: Describe every treatment condition and the control condition in detail. This must include exact stimuli, scripts, images, or question wordings. Specify the mode of delivery (e.g., text, audio, video, in-person) (Gerber et al. 2014).
• Material Availability: Ensure complete treatment materials (vignettes, mailings, software programs) are provided in an appendix for replication. Material availability is not just a reporting requirement; it is infrastructure for cumulative science (Druckman 2022).
• Manipulation Checks: If manipulation checks are used, report their exact wording, placement in the survey flow, and results. Do not selectively exclude respondents who "fail" manipulation checks without pre-specifying this exclusion rule and reporting results both with and without exclusions (Druckman 2022). Place comprehension checks at the end of the survey or after outcome elicitation to avoid signaling the study's purpose (Stantcheva 2023).
• Question Wording Standards: Use item-specific scales rather than agree-disagree, true-false, or yes-no formats to reduce acquiescence bias. Randomize response option order for nominal items; invert order for ordinal items. Separate question stems from response alternatives with a semantic pause for forced-choice items (Stantcheva 2023).
• Soft Launch: Before full deployment, run a small-scale "soft launch" of the complete survey to check for technical issues in the survey flow (loading, display, branching logic), separate from content pretesting. Document any issues discovered and modifications made (Stantcheva 2023).

4. Measurement and Sample Flow

• Variable Definitions: Provide precise definitions for how all primary outcomes, secondary outcomes, and covariates are measured and coded.
• Index Construction: If an index is used, explain exactly how it was constructed.