---
name: jmagly/format-adapter-parquet
source: https://app.decimal.ai/s/jmagly-format-adapter-parquet@1/SKILL.md
source_sha256: 59ee36c160a3
---

# format-adapter-parquet

Convert canonical training example records (`@agentic/code/frameworks/training-complete/schemas/example-record.yaml`) into Apache Parquet files via Apache Arrow — the columnar, compressed, shardable format native to HuggingFace Datasets and high-throughput training pipelines.

## When to Use

- Publishing to HuggingFace Hub (Datasets loads Parquet natively and lazily)
- Large datasets (>100k examples) where JSONL scan cost dominates
- Pipelines that benefit from columnar projection (read only `input.user` + `output.assistant` without materializing metadata)
- Shardable storage for distributed training data loaders

## Parameters

- `<input-glob>` (required) — canonical records (typically the output of `format-adapter-jsonl`)
- `--output <path>` (optional) — default: `.aiwg/training/exports/canonical-<timestamp>.parquet`. Use `<path>/` suffix to shard.
- `--validate-round-trip` (optional) — read Parquet back and verify invariants against input
- `--shard-size <N>` (optional) — rows per shard when producing a directory of Parquet files

## Format Spec

Apache Arrow schema materialized as Parquet — columnar, Snappy-compressed by default, with nested struct columns for `input`, `output`, and `metadata`:

```text
id: string
task_type: string
input: struct<system: string, user: string, context_refs: list<string>, tools_available: list<...>>
output: struct<assistant: string, reasoning_trace: string, tool_calls: list<...>>
metadata: struct<quality_grade: string, license: string, provenance_id: string, created_at: timestamp, domain: list<string>, source_refs: list<string>, difficulty: double, synthetic: bool, synthetic_depth: int32, created_by_agent: string>
```

## Operation

1. **Load canonical records** — parse inputs (commonly JSONL from the identity adapter).
2. **Transform** — build a pyarrow Table with the schema above; coerce timestamps; normalize optional fields to null.
3. **Validate target** — write once to a temp path, reopen, verify column-level schema matches expectation and row count equals input.
4. **Round-trip check** (if `--validate-round-trip`) — reconstruct canonical records from Parquet rows and confirm all `round_trip_invariants` hold.
5. **Write output + log** — finalize single `.parquet` or sharded directory, emit `_metadata` sidecar describing shard layout, append `format-convert` event.

## Round-Trip Invariants

All canonical fields round-trip via nested struct columns. Parquet preserves strong typing and nulls — `id`, `task_type`, `input.user`, `output.assistant`, `metadata.quality_grade`, `metadata.license`, `metadata.provenance_id` all survive verbatim.

## Sidecar Metadata

`<output>.metadata.yaml` (or `_metadata` inside a sharded directory) records: Arrow schema version, compression codec, shard layout, row count per shard, and any columns that were dropped due to cross-record schema inconsistency (should be zero for well-formed inputs).

## Acceptance Criteria

- Output opens cleanly with `pyarrow.parquet.read_table()` and `datasets.load_dataset("parquet", ...)`.
- `--validate-round-trip` reconstructs 100% of canonical invariants.
- Row count equals input record count (minus explicitly rejected schema-invalid inputs).
- `format-convert` event logged with compression ratio and shard count.

## References

- REF-471 — HuggingFace Datasets (Parquet as native storage)
- REF-473 — Apache Arrow + Parquet columnar format
- ADR-022 D7 — canonical + adapter strategy

## Delegation

- `@agentic/code/addons/semantic-memory/skills/memory-log-append/SKILL.md` — logging the `format-convert` event