Training Check

Training Check

Periodically read WandB metrics during training to catch problems early. Do not wait until training finishes to discover it was a waste of GPU time.

Context: $ARGUMENTS

Constants

• WANDB_RUN - Read from project notes or pass as entity/project/run_id.
• CHECK_INTERVAL - Starts at 10 minutes, then gradually increases if consistently healthy: 10 min -> 20 min -> 30 min -> 60 min (cap).
• REVIEWER_MODEL = gpt-5.4 - Used via a secondary Codex agent for ambiguous cases only.

When to Use

• After training is confirmed running (session alive, loss decreasing for the first few steps)
• When the user wants recurring health checks during training
• This skill checks training QUALITY, not process HEALTH. Process health (session alive, GPU utilization) belongs to watchdog-style monitoring.

Workflow

Step 1: Read WandB Metrics

import wandb
api = wandb.Api()
run = api.run("<entity>/<project>/<run_id>")
history = run.history()

If WandB is unreachable (API error, network issue), fall back to reading the log file directly via SSH:

ssh server "tail -100 /path/to/training.log"

Check these signals:

• Loss trend - Is training loss decreasing over the last N steps?
• Eval metrics - Are evaluation metrics improving (or at least not degrading)?
• NaN / Inf - Any NaN or Inf values in loss or gradients?