Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →nvidia/nemo-mbridge-multi-node-slurm
Convert single-node scripts to multi-node Slurm sbatch jobs and debug common multi-node failures. Covers srun-native vs uv run torch.distributed approaches, container setup, NCCL timeouts, OOM sizing for MoE models, and interactive allocation.
Passed 20 of 22 test cases with the skill, and 9 of 22 without it.
and 13 more in the full report
11 fixed9 passed either way2 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 11 test cases the skill fixed. Same model, same prompt, with and without the skill.
I have my interactive allocation and now I'm launching the container shell with `srun --mpi=pmix --no-kill --container-image $IMG --pty bash`. I already specified `--gpus-per-node=8` in my `salloc` command. Do I need to specify GPU flags again in `srun`?
| Checked | Without skill | With skill |
|---|---|---|
| Requires including the `--gpus-per-node` flag in the `srun` command despite it being present in the allocation | Missed | Passed |
The instructions your model reads when the skill loads.
Convert single-node uv run python -m torch.distributed.run commands into multi-node Slurm sbatch scripts with Enroot container support, and debug common multi-node failures.
When converting or debugging Bridge multi-node jobs, answer in this order:
initialize.py: #SBATCH --ntasks-per-node=8 and a direct srun ... uv run python <script> ... launch. Do not wrap these jobs in
python -m torch.distributed.run.RANK, WORLD_SIZE, LOCAL_RANK,
MASTER_ADDR, and MASTER_PORT from SLURM variables during
initialize.py distributed init.HF_HOME, UV_CACHE_DIR, and NEMO_HOME.Failures: to find the first failed rank and nodencclUniqueId, timeout, or crash on rank 0| Approach | ntasks-per-node | Process spawning | Best for |
|---|---|---|---|
| srun-native (preferred) | 8 | Slurm spawns 8 tasks/node | Conversion, inference, Bridge scripts |
| uv run torch.distributed (legacy) | 1 | uv run python -m torch.distributed.run spawns 8 procs/node | MLM pretrain_gpt.py |
Prefer srun-native — simpler, avoids shell escaping issues with TRAIN_CMD. Megatron Bridge auto-derives RANK, WORLD_SIZE, LOCAL_RANK, MASTER_ADDR, MASTER_PORT from SLURM env vars (SLURM_PROCID, SLURM_NTASKS, SLURM_LOCALID, SLURM_NODELIST) via common_utils.py helpers called during initialize.py distributed init, so you never need to set them manually.
Use a shared filesystem for the repository, data, logs, HF_HOME, UV_CACHE_DIR, and NEMO_HOME. NEMO_HOME must not use the container-local default (/root/.cache/nemo) for multi-node SFT/PEFT jobs, because packed-sequence data prepared on node 0 must be visible to the other nodes.
Keep credentials out of sbatch templates and logs. Provide HF_TOKEN, GH_TOKEN, and WANDB_API_KEY through the scheduler environment or a restricted secrets file, and never hardcode token values in the script body. For copy-paste environment and sbatch templates, read references/templates.md.
test cases passed, vs 9 / 22 without
Published skills whose descriptions closely match this one, with their headline benchmark lift.
More skills from the same source and nearby in the registry, with their headline benchmark lift.