Data Wrangling: Universal Access Patterns

Data Wrangling: Universal Access Patterns

Reference for downloading and parsing scientific data from any source. Write and run Python code via Bash for every step.

When to Use

• ToolUniverse tool returned metadata/search results but you need raw or bulk data
• Data is in a format tools don't parse (VCF, h5ad, BAM, SDF, GCT)
• You need a multi-step API workflow (search -> filter -> download -> parse)
• The data source has no ToolUniverse tool at all
• You need thousands of records, not the 10-100 a tool returns

Decision: Tool vs Code

Section A: Format Cookbook

Tabular

import pandas as pd, io

df = pd.read_csv("data.csv")                                # CSV
df = pd.read_csv("data.tsv", sep="\t")                      # TSV
df = pd.read_sas(io.BytesIO(content), format="xport")       # SAS Transport (XPT) — NHANES, CDC
df = pd.read_sas("data.sas7bdat", format="sas7bdat")        # SAS native
df = pd.read_stata("data.dta")                               # Stata — ICPSR, HRS
df = pd.read_parquet("data.parquet")                         # Parquet — MIMIC-IV
df = pd.read_excel("data.xlsx")                              # Excel
df = pd.read_spss("data.sav")                                # SPSS
df = pd.read_fwf("data.dat")                                 # Fixed-width — legacy surveys