Profile-Imaging Skill

Purpose

A dataset decides more of a study than the architecture does, and it decides it first. Before anything is preprocessed, split, or trained, a handful of facts are already true about the data, and each one closes off or opens up a research plan:

• If the target occupies 0.4 % of the volume, accuracy is not a metric — predicting background

everywhere scores 99.6 %.

• If through-plane spacing runs 1.5–8 mm inside a single institution, resampling is not a default to

accept quietly; it is the most consequential preprocessing choice in the study, and it is also the axis along which an external dataset will differ.

• If the directory named imagesTs has no labels, it is not a test set, and the held-out set has to

come from somewhere else — better known before training than after.

• If the organ volume spans 56–502 mL when normal is roughly 100–250, the cohort contains disease

that a subgroup analysis should be pre-specified for, rather than discovered post hoc.

None of that requires a model, a GPU, or an engineer. It requires reading every file once and writing down what is there. This skill does that, and then audits the plan against it.

It is the front door of the model-engineering lane: profile-imaging (describe) → /design-study + /architecture-zoo (decide) → /preprocess-imaging (plan the pipeline) → /model-scaffold (build) → /model-validation → /model-evaluation.

When to use

• You have a dataset and a task, and need to know what the data will and will not support before

committing to a plan.

• You inherited a dataset and need its integrity established (labels intact, splits labelled,

label values as declared) before anyone trains on it.

• You are about to write a Methods section that describes the cohort and its acquisition.

When NOT to use

• Tabular / clinical variables → /generate-codebook (data dictionary) and /clean-data.
• Designing the preprocessing pipeline and auditing it for data-stage leakage →

/preprocess-imaging (it consumes what this skill describes).

• Auditing the train/val/test split table → /model-validation.
• Choosing an architecture → /architecture-zoo. Building the repo → /model-scaffold.
• Held-out metrics, calibration, subgroup results → /model-evaluation then /analyze-stats.

Workflow

Step 1 — profile every case

bash
python3 scripts/profile_imaging_dataset.py \
    --split train:imagesTr:labelsTr \
    --split test:imagesTs \
    --dataset "MSD Task09 Spleen" \
    --declared-labels 0=background,1=spleen \
    --target-label 1 \
    --plan resample=true,reorient=false,loss=dice_ce,metrics=dice+hd95 \
    --out eda/profile.json

One record per case: grid, spacing, orientation, intensity percentiles, the label values actually present, foreground fraction, and target volume in mL. A --split given no label directory is recorded as unlabelled — which is itself a finding.

--target-label on a multi-structure atlas. Foreground defaults to every non-zero index, which is the whole annotated anatomy. Run a single-organ study against a 15-organ atlas and the reported fraction describes the upper abdomen, not the target — measured on AMOS22 that is 3.2 % rather than the spleen's 0.2 %, so the pooled number sits above the 1 % imbalance threshold while the real target sits far below it, and the imbalance verdicts go quiet exactly where the risk is. Naming the target also makes LABEL_EMPTY mean this case has no spleen, which a multi-organ label file otherwise hides behind the other organs. Pass --target-label all for a genuinely multi-class study; leave it out on a multi-structure atlas and the gate raises TARGET_LABEL_UNDECLARED.

Requires nibabel + numpy (it has to open images). The gate below does not.

Step 2 — gate the profile against the declared plan

bash
python3 scripts/check_dataset_profile.py --profile eda/profile.json \
    --out qc/dataset_profile.json --strict

Stdlib-only, so the audit re-runs anywhere the JSON travels. Verdicts:

| Verdict | Severity | Fires when | |---|---|---| | LABEL_SHAPE_MISMATCH | Major | label grid ≠ image grid | | LABEL_EMPTY | Major | a labelled case has zero foreground | | LABEL_VALUE_UNEXPECTED | Major | label values outside the declared set | | TEST_SET_UNLABELLED | Major | a split whose name contains test/held-out/external/eval carries no labels | | ACCURACY_UNDER_IMBALANCE | Major | accuracy is planned while the target is a sliver of the volume | | LABEL_MISSING | Minor | a case in a labelled split has no label file | | SPACING_HETEROGENEOUS | Minor | spacing spans ≥ ratio on an axis and no resampling is declared | | ORIENTATION_MIXED | Minor | >1 orientation code and no reorientation declared | | INTENSITY_SCALE_INCONSISTENT | Minor | some cases sit on the HU scale and others do not | | EXTREME_IMBALANCE | Minor | median foreground below the threshold with no Dice-family loss | | TARGET_LABEL_UNDECLARED | Minor | >1 structure declared but no target named, so foreground pools them all |

The gate flags an undeclared decision, not variability itself. A dataset with 5× spacing spread and two orientation codes passes cleanly once resampling and reorientation are declared — heterogeneity that has been dealt with is not a defect. That distinction is what the challenge card's clean fixture exists to prove.

--spacing-ratio (default 2.0) and --imbalance-frac (default 0.01) are screening defaults, not published cut-points: 2× through-plane spacing changes what a fixed-size patch sees, and 1 % foreground is roughly where plain accuracy stops carrying information. Both are adjustable and both are printed in the output, so a reader knows what was applied.

Step 3 — turn the profile into research decisions

The profile is evidence; the decisions are yours, and the ones worth writing down are:

1. Resampling target — from the spacing distribution, not from a tutorial default. Carry it into

/preprocess-imaging as a declared transform.

1. Loss and metric family — from the foreground fraction. Segmentation reports Dice and a

boundary metric per structure (/model-evaluation); accuracy is not on the list.

1. Pre-specified subgroups — from the clinical spread the profile reveals (target volume,

slice thickness, modality). Pre-specifying them here is what separates a subgroup finding from a post-hoc one.

1. Where the held-out set comes from — especially when the shipped "test" directory is unlabelled.
2. What the cohort cannot support — n, single-source acquisition, absent subgroups. This is the

honest seed of the Limitations paragraph, written before the results can bias it.

Record these in the study record so /design-study, /preprocess-imaging, and eventually /write-paper inherit them rather than re-deriving them.

Outputs

• eda/profile.json — per-case dataset profile (the artifact downstream skills read).
• qc/dataset_profile.json — deterministic audit with verdicts.
• Decision notes for the study record (resampling target, loss/metric family, pre-specified

subgroups, held-out provenance, cohort limitations).

Forbidden

• Reporting a profile figure that was not computed from the files (no remembered spacings, no

assumed label indices — open the labels and look).

• Declaring an audit pass without running the gate.
• Using a split's images as a held-out test set when the profile says it has no labels.
• Reading --spacing-ratio / --imbalance-frac defaults as published thresholds.

Anti-Hallucination

• Never report a profile figure that was not computed from the files. Spacings, label indices,

foreground fractions and organ volumes come from opening every image and label — not from a dataset's documentation, not from what a similar dataset looked like, and not from memory. A dataset's README can be wrong about its own label indices; the labels cannot.

• Never report a profile audit "pass" without running check_dataset_profile.py. The verdicts

are re-derived from the profile JSON by rule and arithmetic; a prose claim that the data "looks fine" is not the audit.

• Never treat an unlabelled split as a test set. If the profile says a split has no labels, no

held-out metric can come from it, however the directory is named.

• Never present --spacing-ratio / --imbalance-frac as published cut-points. They are

screening defaults; the values applied are printed in the output and belong in the Methods.

Validation

bash
python3 scripts/check_dataset_profile.py --profile <profile.json> --strict
bash scripts/check_dataset_profile_challenge/verify.sh   # deterministic, network-free
bash tests/test_dataset_profile.sh