Data Cleaning

Data Cleaning

Purpose

This skill helps economists clean, transform, and prepare datasets for analysis in Python, R, or Stata. It emphasizes reproducibility, proper documentation, and handling common data quality issues found in economic research.

When to Use

• Cleaning raw survey or administrative data
• Merging multiple data sources
• Handling missing values, duplicates, and outliers
• Creating analysis-ready panel datasets
• Documenting data transformations for replication

Instructions

Step 1: Understand the Data

Before generating code, ask the user:

1. What is the data source? (survey, administrative, API, etc.)
2. What is the unit of observation?
3. What are the key variables needed for analysis?
4. Are there known data quality issues to address?

Step 2: Generate Cleaning Pipeline

Create a Stata do-file that:

1. Has a clear header with project info and date
2. Sets up the environment (clear all, set memory, log)
3. Loads and inspects raw data
4. Documents each transformation with comments
5. Creates a codebook for the final dataset

Step 3: Follow Best Practices

• Use assert statements to verify data integrity
• Create labeled variables with label variable
• Use value labels for categorical variables
• Generate a log file for reproducibility