CPU/GPU Performance Discipline

CPU/GPU Performance Discipline

When To Use

• At the beginning of every session (auto-load alongside token-conservation).
• Whenever you plan to build, train, or test anything that could pin CPU cores or GPUs for more than a minute.
• Before retrying a failing command that previously consumed significant resources.

When NOT To Use

• Simple operations with no resource impact
• Quick single-file operations

Required TodoWrite Items

1. cpu-gpu-performance:baseline
2. cpu-gpu-performance:scope
3. cpu-gpu-performance:instrument
4. cpu-gpu-performance:throttle
5. cpu-gpu-performance:log

Step 1: Establish Current Baseline

• Capture current utilization:

  • uptime
  • ps -eo pcpu,cmd | head
  • nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv

  Note which hosts/GPUs are already busy.

• Record any CI/cluster budgets (time quotas, GPU hours) before launching work.

• Set a per-task CPU minute / GPU minute budget that respects those limits.

Step 2: Narrow the Scope

• Avoid running "whole world" jobs after a small fix. Prefer diff-based or tag-based selective testing:
  • pytest -k
  • Bazel target patterns
  • cargo test <module>
• Batch low-level fixes so you can validate multiple changes with a single targeted command.
• For GPU jobs, favor unit-scale smoke inputs or lower epoch counts before scheduling the full training/eval sweep.