Systematic workflow for MoE training optimization in Megatron Bridge, based on the Megatron-Core MoE paper. Covers the Three Walls framework, parallel folding, recompute strategy, dispatcher choice, and CUDA-graph bring-up.
SkillScore91Maintained
Automatic
◆AI Engineer pack
Last verified 8/17/2026
What it changes
Passed 20 of 22 test cases with the skill, and 13 of 22 without it.
Without skill59%
With skill91%
Checks it passed that the model alone missed
The response presents the MoE parallel folding mesh as MoE: ETP x EP x EDP x PP.
The response prioritizes selective recompute before introducing full layer recompute.
The response presents the attention parallel folding mesh as Attention: TP x CP x DP x PP.
and 9 more in the full report
9 fixed11 passed either way0 still failing2 broke
Pass-rate lift
+32 pp
Tokens
+61%
Agent turns
0%
Response time
-48%
Model gemini-3.6-flashMeasured 8/17/2026v220 of 22 cases passed with skill
Tokens, turns and time describe resource use, not a dollar price.
See it in action
One of 9 test cases the skill fixed. Same model, same prompt, with and without the skill.
1 of 9
Prompt
We want to verify whether our 256-GPU tensor, pipeline, and expert parallel config layout will throw shape mismatch or rank allocation errors before reserving the full cluster. What flag allows single-GPU sanity checking of distributed layouts?
What was checked, without and with the skill
Checked
Without skill
With skill
The response specifies using --fake-init-process-group to sanity-check large parallel layouts on a single GPU.
Missed
Passed
case-11 · gemini-3.6-flash · 8/17/2026 · answers are capped and redacted
What's inside
The instructions your model reads when the skill loads.
For operational diagnosis, split the compute-efficiency wall into compute
and host/launch bottlenecks. They need different evidence and different
fixes. MoE tuning is iterative, so use this order:
freeze the measurement contract -> fit -> scale -> profile -> retune -> validate
First Answer Checklist
For MoE optimization workflow prompts, present the response in this order:
Freeze the measurement contract: record the exact model and task,
hardware and topology, container and commits, data and routing semantics,
precision, sequence and batch shape, parallelism, graph scopes, and the
steady-state metric window. Label each candidate as training-equivalent or
benchmark-only.
Fit: make the model memory-feasible first. Use the smallest model
parallelism that fits, prefer selective recompute before full recompute, add
offloading only after recompute and parallelism are insufficient, and use
--fake-init-process-group to sanity-check large layouts.
Scale: maximize DP after the model fits, keep hot communication inside
the fastest interconnect, use PP plus VPP for multi-node scaling, prefer EP
over extra TP for expert layers, and add CP when long context makes attention
memory dominant.
Profile: identify the dominant wall: memory, communication, host