Auto Research

Auto Research

Run iterative NeMo-RL experiments in this repository against the user's stated objective, such as accuracy, reward, throughput, latency, stability, or another recipe-specific metric, with git as the research ledger.

Treat dependencies as ready, but choose the runtime deliberately. Use the recipe's authoritative metric as the source of truth. Keep changes small, reproducible, and simple. Preserve unrelated user work.

Safety: This skill creates git branches, writes files to disk, and executes shell commands including training jobs that may consume GPU resources. Always confirm the campaign plan with the user before creating branches or launching jobs. Do not execute destructive git operations (reset, force-push) or launch compute-intensive jobs without explicit user approval.

Use the nemo-rl-session-memory skill for every auto-research campaign. Start or resume a session record before branching, then checkpoint after forming the plan, before and after meaningful edits or long-running launches, when the user changes direction, and before handoff or final summary.

After context compaction, handoff, disconnect, or a long gap, reload this skill and any companion skills already in use, read the latest nemo-rl-session-memory handoff, and restate the overall objective, stop rules, current branch, and latest result before continuing. Treat follow-up steering as additive unless the user explicitly changes the main objective.

Workflow

1. Inspect the current git state and identify unrelated user changes before branching.
2. Use a shared branch prefix. Prefer a user-provided one; otherwise create a suggestive default such as autoresearch/2026-03-24-dapo-qwen2p5.
3. Read the target recipe, its parents, and the relevant code paths in examples/run_grpo.py, nemo_rl/models/, nemo_rl/algorithms/, nemo_rl/environments/, and docs/. For NeMo-gym recipes, also inspect examples/nemo_gym/ entrypoints, configs, and launch scripts.
4. Translate any user stop rule into explicit values you can monitor, such as the requested number of experiments as target_experiment_count, campaign_deadline, per_experiment_timeout, or target_metric.
5. Verify required data, checkpoints, runtime inputs, and the launcher.
6. Create an untracked TSV log and per-experiment log directory.
7. Run a baseline first on <prefix>/baseline if none exists.

For GPU, CPU-heavy, distributed, or long-running work, choose the execution environment deliberately. Run locally when the current machine has suitable GPUs and capacity; otherwise follow the user's requested environment, use launch-nemo-rl for nrl-k8s/Kubernetes, use the environment's native launcher for Slurm, or clarify with the user before launching. Use CPU-only local runs only for light inspection, dry runs, and short non-GPU checks.

If the user mentions Brev, or if /home/ubuntu/RL exists and /ephemeral is available as a volume, treat the machine as a Brev instance and use nemo-rl-brev-etiquette before creating experiment directories, caches, logs, checkpoints, or authenticated runtime state.

Branching

• Put every experiment on its own branch under the shared prefix.
• Keep every branch, even for failed or weak ideas.
• Put at least one commit on each branch for the hypothesis.
• Add follow-up fix commits on the same branch when a rerun is justified.
• Never stash, reset, or overwrite unrelated user changes silently. If dirty files overlap the experiment, use a separate worktree or ask before proceeding.

See references/git-workflow.md for the exact pattern.

Loop

1. Pick one concrete hypothesis.
2. Create a branch such as autoresearch/2026-03-24-dapo-qwen2p5/prompt-compact-schema.