Choose Big Data Solution

Choose Big Data Solution

Help users and agents select the right MATLAB tool for large tabular data in file-based formats (CSV, Parquet, delimited text, spreadsheets, MDF). The skill encodes a decision flowchart — recommend one clear path, not a menu of options.

When to Use

• User or agent has large tabular data in file-based formats (CSV, Parquet, delimited text, spreadsheets, MDF)
• User or agent says data is "large", "big", or "huge" (even without a specific size)
• User or agent hits an out-of-memory error with readtable, parquetread, or similar
• User or agent has multiple tabular files to process as a single dataset
• User or agent has multiple tabular files to process independently (per-file)
• User or agent asks how to scale up an existing workflow on tabular file data
• User or agent asks about datastores, tall arrays, or mapreduce
• User or agent needs to build a custom datastore class for a proprietary or non-standard format
• User or agent has large XML, JSON, HTML, or Word document files — readtable supports these formats but the built-in datastores do not. Scaling these requires a custom datastore (see "Formats Without a Built-in Datastore" and "Custom Datastores" sections below)

When NOT to Use

• Single file that fits in memory (file-size-to-RAM ratio < 0.5) — see Pre-Flight Check and matlab-import-export-data skill
• User or agent is working with MAT files — matfile provides partial I/O for large .mat files, different workflow
• User or agent is working with databases (SQL, ODBC) — use Database Toolbox. See the matlab-read-database and matlab-use-duckdb skills in the reporting-and-database-access category
• User or agent needs GPU acceleration — different domain
• Deep learning training workflows — datastore is correct for data loading, but use minibatchqueue for batching (not tall or transform)

Pre-Flight Check

ALWAYS run this check BEFORE recommending datastore or tall array patterns. Estimate the file-size-to-available-RAM ratio (accounting for in-memory expansion of the file format).