Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →nvidia/launch-nemo-rl
Playbook for launching, monitoring, stopping, and debugging NeMo-RL recipes on a Kubernetes cluster via the nrl-k8s CLI. Covers ephemeral vs long-lived RayCluster modes, iterating on runs, and debugging hung or failed training jobs.
Passed 22 of 23 test cases with the skill, and 7 of 23 without it.
and 26 more in the full report
17 fixed6 passed either way0 still failing1 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 16 test cases the skill fixed. Same model, same prompt, with and without the skill.
I am iterating on a disaggregated NeMo-RL recipe where the gym environment and generation daemon services are already running healthily on the cluster. I only want to re-submit the training job without touching or re-deploying the live daemons. Which CLI option prevents re-submitting daemons?
| Checked | Without skill | With skill |
|---|---|---|
| Passes the --skip-daemons flag to nrl-k8s run. | Missed | Passed |
The instructions your model reads when the skill loads.
This is the playbook for the nrl-k8s CLI at infra/nrl_k8s/. Follow it when the user asks to launch / iterate / debug a NeMo-RL recipe on a Kubernetes cluster. Verify current state (kubectl, git log, the recipe + infra files) before acting — the cluster is shared and the cost of a wrong action is high.
There is a single top-level submission command: nrl-k8s run. It has two lifecycle modes.
| Mode | Invocation | When to use | Cluster after? |
|---|---|---|---|
| Ephemeral (default) | nrl-k8s run | One-shot. KubeRay applies a RayJob, runs, tears the cluster down. Best for most runs. | No (auto) |
| Long-lived | nrl-k8s run --raycluster | Dev loop. Reuses a matching live cluster, applies if absent, warns + reuses on drift (pass --recreate to replace). Then submits daemons and training. First-choice for iteration. | Yes |
Ask: Do I need this cluster after the run? If yes, use --raycluster. Otherwise use the default (ephemeral).
The rest of the CLI is observability / stage-by-stage control:
| Command | Purpose |
|---|---|
nrl-k8s check | Validate a recipe + infra pair; optionally write the fully-resolved manifests (-o). |
nrl-k8s status | Per-role RayCluster state, head pod phase, worker pod phases, daemon job status. |
nrl-k8s cluster up/down/list/dashboard | Manage RayClusters independently of a run (e.g. render a manifest with --dry-run). |
nrl-k8s job list/logs/stop | Observability over Ray Jobs already submitted to a role's cluster. |
nrl-k8s logs | Tail a role's pod / daemon logs without needing a submission id. |
Every launch takes two files. Pass the infra with --infra, not merged inline:
nrl-k8s run infra/nrl_k8s/examples/<recipe>.yaml \
--infra infra/nrl_k8s/examples/<recipe>.<profile>.infra.yaml
qwen3_30b_math_8n_4gpu.yaml) — NeMo-RL config: model, GRPO/SFT knobs, cluster.{gpus_per_node,num_nodes}. Uses defaults: to inherit from examples/configs/recipes/llm/....*.<profile>.infra.yaml) — K8s/Ray shape: namespace, image, service account, RayCluster spec under kuberay:, optional Deployments under deployments:, submit.submitter, launch.{mode,codeSource,codePath,entrypoint}. Pair names follow <recipe>.<profile>[.prod].infra.yaml where <profile> names the hardware target (e.g. gb300).Example pairs in infra/nrl_k8s/examples/ — read the neighbouring files to see the current conventions for the target profile.
test cases passed, vs 7 / 23 without
More skills from the same source and nearby in the registry, with their headline benchmark lift.