Reproducible Pipelines

Reproducible Pipelines

Reference for building reproducible research pipelines: from project directory structure to automated workflows to journal-ready replication packages. Every computational result should be regenerable from raw data by running a single command.

When to Use This Skill

Use when the user is:

• Setting up a new empirical research project
• Building or debugging a Makefile/Snakemake/DVC pipeline
• Preparing a replication package for journal submission
• Managing computational environments (conda, Docker, renv)
• Tracking data provenance or versioning large datasets
• Debugging "works on my machine" reproducibility failures

Skip when:

• The task is about estimation methodology (use causal-inference or structural-modeling skill)
• The task is git workflow management (see workflows-work/references/worktree-patterns.md)
• The task is about orchestrating Claude agents (see slfg/references/orchestration-patterns.md)

Where to Start

• New project? Start with Directory Structure below
• Adding a workflow manager? Jump to Workflow Managers (Make / Snakemake / DVC)
• Preparing for submission? Jump to Pre-Submission Checklist

Project Directory Structure

Use a standardized layout from the start. This is the structure expected by most replication reviewers:

project/
├── README.md                 # Master documentation (how to replicate)
├── Makefile                  # Or Snakefile — single entry point
├── environment.yml           # Conda environment (or requirements.txt)
├── data/
│   ├── raw/                  # Original, immutable data files
│   │   └── README.md         # Data sources, access instructions, citations
│   ├── intermediate/         # Cleaned/transformed data (gitignored, regenerable)
│   └── final/                # Analysis-ready datasets (gitignored, regenerable)
├── code/
│   ├── 01_clean.py           # Data cleaning