HuggingFace Datasets

HuggingFace Datasets

Overview

The datasets library by HuggingFace provides memory-efficient, Apache Arrow-backed access to tens of thousands of public datasets and your own local/remote data. It supports streaming for datasets that don't fit in RAM and integrates directly with transformers tokenizers and training loops.

When to Use

Use this skill when:

• Loading datasets from the HuggingFace Hub or local files (Parquet, JSONL, CSV, Arrow).
• Streaming datasets too large for memory.
• Applying preprocessing, tokenization, or filtering at scale with .map() and .filter().
• Pushing curated datasets to the Hub for sharing or versioning.
• Interleaving or concatenating multiple datasets.

Do not use for:

• Small in-memory pandas DataFrames with < 100K rows — use pandas directly.
• Model inference — use transformers or vllm skills.
• Embedding computation — use embedding-analysis skill.

Installation

uv pip install datasets pyarrow huggingface_hub

Quick Start

Loading

from datasets import load_dataset, load_from_disk

# From the Hub
dataset = load_dataset("imdb", split="train")

# From local files
dataset = load_dataset("parquet", data_files="data/*.parquet", split="train")

# From disk (pre-saved)