Vaex DataFrames

Vaex DataFrames

Overview

Vaex is a high-performance Python library for lazy, out-of-core DataFrame operations on datasets too large to fit in RAM. It processes over a billion rows per second using memory-mapped files and lazy evaluation, enabling interactive exploration and analysis without loading data into memory.

When to Use

• Processing tabular datasets larger than available RAM (10 GB to terabytes)
• Fast statistical aggregations on massive datasets (mean, std, quantiles at billion-row scale)
• Creating visualizations (heatmaps, histograms) of large datasets without sampling
• Building ML preprocessing pipelines (scaling, encoding, PCA) on big data
• Converting between data formats (CSV to HDF5/Arrow for fast repeated access)
• Feature engineering with virtual columns that consume zero additional memory
• Working with astronomical catalogs, financial time series, or large scientific datasets
• For in-memory speed on data that fits in RAM, use polars instead
• For distributed multi-node computing, use dask instead

Prerequisites

pip install vaex
# Optional extras:
pip install vaex-hdf5          # HDF5 support (recommended)
pip install vaex-arrow          # Apache Arrow support
pip install vaex-ml             # Machine learning transformers
pip install vaex-viz            # Visualization support
pip install vaex-jupyter        # Jupyter widget support
pip install s3fs gcsfs adlfs    # Cloud storage (S3, GCS, Azure)

Requires Python 3.7+. HDF5 and Arrow formats provide instant memory-mapped loading; CSV requires conversion for optimal performance.

Quick Start

import vaex
import numpy as np

df = vaex.from_arrays(