Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →nvidia/cupynumeric-parallel-data-load
Load a sharded, on-disk dataset (sharded .npy, Parquet/Arrow, raw binary, sharded HDF5, custom layouts) into a distributed cuPyNumeric ndarray via a manual partition + leaf @task launch with CPU/OMP/GPU variants.
Passed 17 of 21 test cases with the skill, and 5 of 21 without it.
and 19 more in the full report
13 fixed5 passed either way4 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 12 test cases the skill fixed. Same model, same prompt, with and without the skill.
I need to load a single HDF5 dataset file into a cuPyNumeric array in a Legate cluster application. Should I write a manual `partition_by_tiling` task launch with `h5py` calls inside leaf tasks, or does Legate provide a direct HDF5 loader?
| Checked | Without skill | With skill |
|---|---|---|
| Recommends using legate.io.hdf5.from_file instead of a custom task launch. | Missed | Passed |
| Identifies single-file HDF5 loading as a built-in Legate loader capability. | Passed | Passed |
The instructions your model reads when the skill loads.
Why this skill exists. cupynumeric mirrors NumPy's array API, including
cupynumeric.load for a single .npy file. Beyond that, file loading lives
in Legate, not cupynumeric:
| Format | Built-in loader |
|---|---|
Single .npy | cupynumeric.load(path) (NumPy-API parity) |
| HDF5 (single file) | legate.io.hdf5.from_file / from_file_batched |
| Sharded multi-file (any format), Parquet/Arrow, raw binary, custom layouts | No built-in loader — this skill. |
This skill shows the canonical way to fill the gap in the last row:
write a Legate Python task that calls the third-party reader the
format needs (h5py, pyarrow, np.memmap, ...) inside the
task body, and let Legate distribute the reads across GPUs / nodes.
For the formats with a built-in loader, prefer it unless you need a
custom in-task body (mmap-based loader, format-specific decoder,
sidecar metadata, partial / sharded reads).
Canonical pattern: manual partition + manual task launch, sized to
the machine, not the files. Only axis 0 is sharded; trailing axes
ride along inside each tile. Per-shard row counts may differ across
files (only dtype and trailing axes must match); the launch fills
every available processor regardless of how many files there are.
.npy is the worked example because the header carries shape and
dtype on disk, but the skeleton applies to any format with cheap
range/slice reads (raw binary, HDF5, Parquet/Arrow — see "Other
formats" below). Reference implementation:
assets/examples/parallel_npy_load.py.
This skill is purely about loading — it assumes the data is already
laid out on a shared filesystem in some predictable, indexable way.
Producing those files is out of scope (the example ships a write
subcommand for convenience, but real users bring their own).
The worked example assumes one specific layout:
test cases passed, vs 5 / 21 without
More skills from the same source and nearby in the registry, with their headline benchmark lift.