Dataset Discovery

When to Use

• User asks "find me data about X" or "where can I get data on Y"
• User wants to analyze a relationship between variables
• User needs specific study designs (longitudinal, cross-sectional, experimental)
• User asks about specific surveys or cohorts

Step 1: Understand What the Research Question Requires

Before searching, determine the minimum data requirements:

Study design needed:

• "Does X predict CHANGES in Y over time?" → longitudinal (same people measured repeatedly). Cross-sectional data CANNOT answer this — don't settle for it.
• "Is X associated with Y?" → cross-sectional is sufficient (one-time measurement)
• "Does intervention X cause outcome Y?" → experimental (clinical trial with controls)
• "What genes/proteins are involved in X?" → omics (sequencing, expression, proteomics)

Variables needed:

• List the specific exposure, outcome, and confounder variables
• For each variable, note the measurement type (continuous, categorical, biomarker vs self-report)
• Identify minimum confounders needed (age, sex are almost always required; domain-specific confounders depend on the question)

Population needed:

• Age range, geography, clinical status, sample size requirements
• Power analysis: to detect a small effect (r=0.1), you need ~800 subjects at 80% power

Step 2: Search Strategy

Search from broadest to most specific. Use find_tools to discover available dataset search tools — don't rely on memorized tool names.

Layer 1 — Cross-repository search (cast wide net): Search tools that index datasets across thousands of repositories. These find datasets you didn't know existed.

• Search by: research topic keywords, variable names, population descriptors
• Look for: DOI-registered datasets, repository listings, government data portals

Layer 2 — Domain-specific repositories: Search repositories specialized for your data type.

• Health surveys: CDC, NHANES (search by variable name, not topic keywords)
• Genomics: SRA, ENA, ArrayExpress, GEO
• Proteomics: PRIDE, MassIVE
• Metabolomics: MetaboLights, Metabolomics Workbench
• Clinical: ClinicalTrials.gov (for trial data with results)

Layer 3 — Literature-based discovery: Many datasets aren't in any repository — they're described in paper methods sections.

• Search PubMed/EuropePMC for papers that analyzed the relationship you're interested in
• Read their methods: "We used data from DATASET NAME]" tells you exactly what exists
• Check supplementary materials for deposited data (GEO/SRA accession numbers)
• This is often the MOST effective strategy for finding niche datasets

Step 3: Evaluate Dataset Fitness

For each candidate dataset, assess these dimensions:

Variables:

• Does it contain your SPECIFIC exposure and outcome variables?
• Are they measured the way you need? (biomarker vs self-report, continuous vs categorical)
• Are key confounders available? (missing confounders = biased analysis)

Design match:

• If you need longitudinal: does it follow the SAME individuals over time? How many waves? What's the follow-up interval?
• Beware: "repeated cross-sections" (different people each wave) are NOT longitudinal
• If you need experimental: is there a proper control group? Randomization?

Sample:

• Is the sample large enough for your analysis? (logistic regression needs ~10 events per predictor)
• Does the population match? (age range, geography, clinical characteristics)
• Are there subgroups you need? (stratified by sex, race, disease status)

Access:

• Publicly downloadable (best) vs registration required (days) vs collaboration agreement (months) vs restricted (may be impossible)
• Data format: CSV/TSV (easy), XPT/SAS (need conversion), proprietary database (may need special software)

Quality:

• Is it from a well-known study with published methods? (NHANES, HRS, UK Biobank = high quality)
• Has it been used in peer-reviewed publications? (indicates data is usable)
• What's the response rate / missingness pattern?

Step 4: Download and Analyze

Don't stop at finding datasets — download and analyze them. Write and run Python code via Bash. Never describe what you "would do" — execute it.

Data Loading Cookbook

Choose the loader that matches your data source. When unsure of the format, download a small sample first and inspect.

python
import requests, io, pandas as pd

# --- Tabular files (most common) ---
df = pd.read_csv("data.csv")                                # CSV / TSV (use sep="\t" for TSV)
df = pd.read_excel("data.xlsx")                              # Excel
df = pd.read_stata("data.dta")                               # Stata
df = pd.read_sas("data.xpt", format="xport")                # SAS transport (XPT)
df = pd.read_sas("data.sas7bdat", format="sas7bdat")        # SAS native
df = pd.read_parquet("data.parquet")                         # Parquet
df = pd.read_json("data.json")                               # JSON (records or columnar)
df = pd.read_fwf("data.dat")                                 # Fixed-width (some legacy surveys)

# --- Download from URL first, then parse ---
resp = requests.get(url, timeout=120)
content = resp.content
# Detect format from URL or content header
if url.endswith(".XPT") or url.endswith(".xpt"):
    df = pd.read_sas(io.BytesIO(content), format="xport")
elif url.endswith(".csv") or url.endswith(".csv.gz"):
    df = pd.read_csv(io.BytesIO(content))
elif url.endswith(".tsv") or url.endswith(".tsv.gz"):
    df = pd.read_csv(io.BytesIO(content), sep="\t")
elif url.endswith(".json"):
    df = pd.read_json(io.BytesIO(content))
else:
    # Try CSV first, then inspect
    df = pd.read_csv(io.BytesIO(content))

# --- REST API pagination (common for GDC, ClinicalTrials.gov, etc.) ---
import json
all_records = []
offset = 0
while True:
    resp = requests.get(f"{api_url}?offset={offset}&limit=100", timeout=30)
    batch = resp.json().get("data", [])
    if not batch:
        break
    all_records.extend(batch)
    offset += len(batch)
df = pd.DataFrame(all_records)

Merge, Clean, Analyze

python
# Merge multiple files on participant/sample ID
merged = df1.merge(df2, on="id_col", how="inner")

# Filter population
subset = merged[(merged["age"] >= 60) & (merged["age"] <= 80)].copy()

# Handle missing values
missing_pct = subset.isnull().mean() * 100
print("Missing % per variable:\n", missing_pct[missing_pct > 0].sort_values(ascending=False))
subset = subset.dropna(subset=["exposure_var", "outcome_var"])

# Quick regression
import statsmodels.formula.api as smf
model = smf.ols("outcome ~ exposure + age + sex", data=subset).fit()
print(model.summary())

# Visualization
import matplotlib.pyplot as plt
plt.scatter(subset["exposure"], subset["outcome"], alpha=0.3)
plt.xlabel("Exposure"); plt.ylabel("Outcome")
plt.savefig("/tmp/scatter.png", dpi=150, bbox_inches="tight")

Always run the code and report actual numbers (β, p-value, CI, N).

Step 5: Report Honestly

Structure the report as:

1. Best available dataset — name, what it contains, access method, key limitation
2. Analysis results — actual statistics (β, p-value, CI, N) from running the code
3. Alternative datasets — ranked by fitness, with tradeoffs
4. What CANNOT be answered — if no dataset matches the study design needed, say so clearly
5. Recommended next steps — apply for access to longitudinal data, replicate in other cohorts

Critical honesty rules:

• Never claim a dataset answers a temporal question if it's cross-sectional
• Distinguish "data exists but needs registration" from "data doesn't exist"
• Report actual computed statistics, not hypothetical analyses
• State the strongest analysis possible with available data, even if it's weaker than what was asked

LOOK UP, DON'T GUESS

Never assume a dataset exists — search for it. Never assume access is public — check. Never assume variables are measured the way you need — verify the codebook.