Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →nvidia/mcore-run-on-slurm
How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.
Passed 25 of 25 test cases with the skill, and 17 of 25 without it.
and 7 more in the full report
8 fixed17 passed either way0 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 8 test cases the skill fixed. Same model, same prompt, with and without the skill.
We are enabling overlap_moe_expert_parallel_comm for a Mixture-of-Experts Megatron-LM pretraining run on our SLURM cluster. What numerical value should we set for CUDA_DEVICE_MAX_CONNECTIONS in our environment configuration?
| Checked | Without skill | With skill |
|---|---|---|
| Recommends setting CUDA_DEVICE_MAX_CONNECTIONS to 32. | Missed | Passed |
The instructions your model reads when the skill loads.
For text-only SLURM setup questions, answer with these constants before the full script:
cd there in the
script before launching training.srun task per node and launch workers with
uv run python -m torch.distributed.run, not bare torchrun.MASTER_ADDR from
scontrol show hostnames "$SLURM_JOB_NODELIST" | head -n1, set MASTER_PORT,
NNODES=${SLURM_NNODES}, GPUS_PER_NODE=<GPUS_PER_NODE>, and
WORLD_SIZE=$((NNODES * GPUS_PER_NODE)).--nnodes, --nproc-per-node, --node-rank, --master-addr, and
--master-port to torch.distributed.run.CUDA_DEVICE_MAX_CONNECTIONS: pre-Blackwell Hopper/Ampere with TP>1 or CP>1
and non-FSDP uses 1; Blackwell/GB200 does not need it; Torch-FSDP2 or
Megatron-FSDP must not use 1; overlap_moe_expert_parallel_comm uses 32.uv installed; run uv sync --extra training --extra dev (or --extra lts) on the worktree once before submission so the .venv is materialized and visible to every node.Save as run_megatron.slurm in the worktree:
#!/bin/bash
#SBATCH --job-name=megatron
#SBATCH --account=<SLURM_ACCOUNT>
#SBATCH --partition=<SLURM_PARTITION>
#SBATCH --nodes=<NODES>
#SBATCH --ntasks-per-node=1
#SBATCH --gpus-per-node=<GPUS_PER_NODE>
#SBATCH --time=<HH:MM:SS>
test cases passed, vs 17 / 25 without
Published skills whose descriptions closely match this one, with their headline benchmark lift.
More skills from the same source and nearby in the registry, with their headline benchmark lift.