Dask — Parallel & Distributed Computing

Dask — Parallel & Distributed Computing

Overview

Dask is a Python library for parallel and distributed computing that scales familiar pandas/NumPy APIs to larger-than-memory datasets. It provides five main components (DataFrames, Arrays, Bags, Futures, Schedulers) and scales from single-machine multi-core to multi-node HPC clusters.

When to Use

• Processing datasets that exceed available RAM (10 GB–100 TB)
• Parallelizing pandas or NumPy operations across multiple cores
• Processing multiple files efficiently (CSV, Parquet, JSON, HDF5, Zarr)
• Building custom parallel workflows with task dependencies
• Distributing workloads across HPC clusters (SLURM, Kubernetes)
• Streaming/ETL pipelines for unstructured data (logs, JSON records)
• For in-memory single-machine speed: use polars instead
• For out-of-core single-machine analytics: use vaex instead

Prerequisites

pip install dask[complete]        # All components
pip install dask[dataframe]       # DataFrames only
pip install dask[distributed]     # Distributed scheduler + dashboard
pip install dask-jobqueue          # HPC cluster integration (SLURM, PBS)

Core API

1. DataFrames — Parallel Pandas

import dask.dataframe as dd

# Read multiple files as a single DataFrame
ddf = dd.read_csv('data/2024-*.csv')
ddf = dd.read_parquet('data/', columns=['id', 'value', 'category'])

# Operations are lazy until .compute()
filtered = ddf[ddf['value'] > 100]
result = filtered.groupby('category').agg({'value': ['mean', 'sum']}).compute()