Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →nvidia/nemo-automodel-distributed-training
Guide for selecting and configuring distributed training strategies in NeMo AutoModel, including FSDP2, Megatron FSDP, DDP, and parallelism settings.
Passed 22 of 22 test cases with the skill, and 9 of 22 without it.
and 20 more in the full report
13 fixed9 passed either way0 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 13 test cases the skill fixed. Same model, same prompt, with and without the skill.
Before enabling pp_size: 2 in the recipe YAML, what specific attribute must exist on the Hugging Face model class for AutoPipeline to shard layers?
| Checked | Without skill | With skill |
|---|---|---|
| Identifies _pp_plan as the required model class attribute. | Missed | Passed |
The instructions your model reads when the skill loads.
NeMo AutoModel uses PyTorch-native distributed training.
All parallelism is orchestrated through a single MeshContext object that
holds device meshes, strategy configs, and axis names.
For conceptual distributed-training questions, answer directly from the quick patterns in this skill without inspecting the repository. Start with the strategy choice, then list only the YAML fields and constraints relevant to the question.
Use direct action verbs in the final answer: recommend the strategy, show the minimal YAML, state the sizing constraint, and name the unsupported strategies. Do not discuss model onboarding, recipes, Slurm, SkyPilot, or checkpointing unless the user asks.
Recommend strategy: fsdp2. Mention tp_size, pp_size, cp_size,
ep_size, and the pipeline sub-config. State that dp_size is inferred from
world_size / (tp_size * pp_size * cp_size).
distributed:
strategy: fsdp2
tp_size: 8
pp_size: 4
cp_size: 1
ep_size: 1
pipeline:
pp_schedule: interleaved1f1b
pp_microbatch_size: 1
test cases passed, vs 9 / 22 without
More skills from the same source and nearby in the registry, with their headline benchmark lift.