Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →nvidia/nemo-mbridge-resiliency
Resiliency features in Megatron Bridge including fault tolerance, straggler detection, in-process restart, preemption, and re-run state machine.
Passed 19 of 20 test cases with the skill, and 4 of 20 without it.
and 30 more in the full report
17 fixed4 passed either way1 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 15 test cases the skill fixed. Same model, same prompt, with and without the skill.
When calc_ft_timeouts is set to True in Megatron Bridge fault tolerance configuration, where are the computed optimal section timeouts persisted?
| Checked | Without skill | With skill |
|---|---|---|
| Identifies ft_state.json as the output file | Missed | Passed |
The instructions your model reads when the skill loads.
Stable docs: @docs/training/resiliency.md, @docs/training/checkpointing.md Card: @skills/nemo-mbridge-resiliency/card.yaml
from megatron.bridge.recipes.run_plugins import FaultTolerancePlugin
import nemo_run as run
task = run.Script(...)
run_plugins = [
FaultTolerancePlugin(
enable_ft_package=True,
calc_ft_timeouts=True,
num_in_job_restarts=3,
num_job_retries_on_failure=2,
initial_rank_heartbeat_timeout=1800,
rank_heartbeat_timeout=300,
)
]
run.run(task, plugins=run_plugins, executor=executor)
| Plugin parameter | Default | Description |
|---|---|---|
num_in_job_restarts | 3 | Max restarts within same job |
num_job_retries_on_failure | 2 | Max new job launches on failure |
initial_rank_heartbeat_timeout | 1800 | First heartbeat timeout (seconds) |
rank_heartbeat_timeout | 300 | Subsequent heartbeat timeout (seconds) |
from megatron.bridge.training.config import FaultToleranceConfig
test cases passed, vs 4 / 20 without
More skills from the same source and nearby in the registry, with their headline benchmark lift.