Launch: pre-flight checklist for long ML training jobs

Long training jobs are expensive to fail. A 12-hour run that crashes on epoch 3 from a missing dataset path or a default workers=8 against an NFS mount is a full day lost. This skill walks five quick checks before you commit the GPUs.

The agentic Stop hook in this plugin will route here from reason when an assistant tries to launch a run without going through the checklist.

When to run

The user just asked to:

• launch / kick off / start / fire up a training run
• restart a run that died
• kill a current run (also runs the cleanup half of the checklist)
• review a launch command before submitting

Or the user is about to run any of: python train.py, accelerate launch, torchrun, deepspeed, sbatch train.sh, tmux new-session ... python ... train, wandb sweep.

The checklist

1. Config diff against a reference run

The most expensive failure is launching with the wrong knobs. Before starting:

bash
find configs/ recipes/ experiments/ -maxdepth 3 \( -name '*.yaml' -o -name '*.yml' -o -name '*.json' -o -name '*.toml' \) -mtime -30 2> /dev/null | head

Pick the most-recently-modified config that resembles the intended run (same model family, same task). Diff against the intended config:

bash
diff -u configs/baseline_v1.yaml configs/intended.yaml

Walk every diff line. For each, ask: _is this difference intentional and motivated, or is it a stale default I forgot to set?_ Common silent regressors:

• num_workers / dataloader workers (default in many repos is 8: wrong on NFS)
• batch_size (per-device vs global mismatch under DDP)
• learning_rate (linearly scaled with batch size; if batch changed, lr should too)
• optimizer betas / weight decay (paper-default vs framework-default)
• mixed_precision (fp16 vs bf16 matters for some models)
• gradient_accumulation_steps
• seed (still set if you care about reproducibility)

If no reference exists in this project, ask the user to point at one. Do not launch with framework defaults alone.

2. Run name discipline

The run name will live in wandb / neptune / checkpoint dirs / status reports for the rest of its life. It must describe the experiment in plain English without internal codes:

• bad: run-1, wave-2, cs-ad, phase2-internal
• good: 7src-fastvit-s-featmap-mlp-dinov3, coco-baseline-bs256-lr3e-4, swin-t-imagenet-distill-from-vit-l

The pattern: <dataset/task>-<model>-<key-config>-<distinctive-recipe-piece>. If you can't describe the experiment from the name in one sentence, the name is wrong. The Stop hook flags any run reference that uses session-local labels.

3. Path verification

Before launching, every path the run depends on must be confirmed to exist:

bash
# Dataset path
ls -la /path/to/dataset | head

# Pretrained checkpoint (if loading)
ls -la /path/to/checkpoint.pt

# Output directory parent (must exist; the run dir will be created)
ls -la /path/to/runs/

# Config file
cat configs/intended.yaml | head

Never trust a path that was recalled from memory. The destructive_path_guard.sh hook will already block obvious cases for rm/mv, but the launch path needs the same scrutiny, a run started with a nonexistent dataset path crashes 30 minutes in instead of immediately.

4. Monitoring setup

Auto-detect the experiment tracker:

• WANDB_API_KEY set or wandb import in the launcher → wandb
• NEPTUNE_API_TOKEN set → neptune
• MLFLOW_TRACKING_URI set or mlflow in launcher → mlflow
• presence of runs/ or lightning_logs/ → tensorboard
• none of the above → ask the user; "no monitoring" is rarely the right answer for a multi-hour run

Confirm the run will appear under the right project / entity / experiment-name. Confirm any tags / groups for cohort comparison are set.

5. ETA in your local timezone

Estimate wall-clock duration: epochs × seconds-per-epoch / 3600 = hours. State the ETA in your local TZ (the system's TZ, which the timezone_scrub.sh hook validates against). If the run will straddle a meeting / sleep / OOO window, decide whether to defer or split.

Restart and kill cleanup

If this is a restart of a previously-failed run, or a kill before launching a replacement, purge stale artifacts in this exact order:

1. Local checkpoint dir on the launching machine: rm -rf /local/runs/<run-name> (verify path first; the destructive_path_guard.sh will warn).
2. Remote artifact dir on the cluster / NFS / object store: rm -rf /remote/runs/<run-name> (or equivalent).
3. Experiment tracker run: delete via the tracker's API (wandb api.run(...).delete(), neptune run.stop() + delete via UI, etc.). Stale tracker runs corrupt later comparisons.
4. Scheduler reservation: cancel the SLURM job (scancel <jobid>), the lambda labs reservation, the cron entry, etc. Runs that "killed but the GPUs are still allocated" are a recurring waste.

Skipping any of these creates ghost state that will confuse the next launch or the next comparison.

Output

When the user invokes this skill, walk the five checks (or three checks + cleanup, if killing) and report which passed and which failed. Block the launch on any failure unless the user explicitly waives the check.

For a clean launch, end with the launch command itself in a fenced block, ready to copy.