---
name: wshobson/grpo-rlvr-training
source: https://app.decimal.ai/s/wshobson-grpo-rlvr-training@1/SKILL.md
source_sha256: 34195072e36d
---

# GRPO & RLVR Training

This skill assumes `finetuning-method-selection`
already routed here because the target behavior
has a verifiable pass/fail signal — not
demonstrations (`lora-qlora-recipes`) or
preference pairs (`preference-optimization`).
What follows is when RL is the right tool, the
reference recipe, the mandatory reward-inspection
gate, and how to pick a GRPO variant when the
base recipe misbehaves.

**Input:** a routing decision (RLVR via GRPO)
plus a verifier (code executor, test suite,
schema checker, or grader) for the target task.
**Output format:** a validated GRPO config — the
kwarg values in `references/grpo-memory.md` and
the reward functions in
`references/reward-functions.md`, not free-form
advice — that `llm-finetuning-training-engineer`
consumes directly.

## When RL Applies

GRPO+RLVR only pays off when task success is
**algorithmically checkable** — a unit test
passes, a parser accepts the output, a tool call
matches an expected schema, a math answer matches
a ground truth. If grading the output requires
human judgment or a subjective rubric, that's an
eval-harness and judge-calibration problem first
— see `eval-harness-first` — not a reason to skip
straight to RL.

Before opening a GRPO run, confirm the model can
**sometimes** succeed on the target task already.
RL sharpens an existing capability by reweighting
toward the samples that already work; it does not
install a capability from zero.

- **The model never succeeds, even at low
  temperature across many samples:** the gap is
  format or task understanding, not policy
  refinement. Route back to SFT first
  (`lora-qlora-recipes`) and only return to this
  skill once the base success rate is nonzero.
- **The model succeeds sometimes,
  inconsistently:** this is the GRPO sweet spot —
  proceed to The Recipe below.

The standing rule for the whole plugin: **DPO for
taste, GRPO for reasoning.** If the signal is a
preference between two acceptable outputs, that's
`preference-optimization`, not this skill.

## The Recipe

The reference recipe is TRL's `GRPOTrainer` with
vLLM-backed generation:

```python
from trl import GRPOConfig, GRPOTrainer

grpo_args = GRPOConfig(
    output_dir="./outputs-grpo",
    use_vllm=True,
    vllm_mode="colocate",       # single GPU; "server" for multi-GPU
    num_generations=8,          # floor — fewer starves the group-relative baseline
    learning_rate=5e-7,         # settled range for GRPO
    beta=0.01,                  # KL coefficient vs the reference policy
    per_device_train_batch_size=8,
    gradient_accumulation_steps=4,
    bf16=True,
    logging_steps=10,
    seed=3407,
)

trainer = GRPOTrainer(
    model=SFT_CHECKPOINT,
    args=grpo_args,
    reward_funcs=[format_reward, correctness_reward],   # references/reward-functions.md
    train_dataset=prompts,       # prompt-only — GRPO generates its own completions
    processing_class=tokenizer,
)

trainer.train()
```

- **`vllm_mode="colocate"`** runs generation and
  training on the same GPU — the default for a
  single-GPU box.
- **`vllm_mode="server"`** points at a separate
  vLLM server process and is the multi-GPU path —
  generation and training don't compete for the
  same device.
- **`num_generations` ≥ 8** is a floor, not a
  suggestion: GRPO's advantage estimate is
  relative to the group mean, and fewer than 8
  samples per prompt produces a noisy baseline.
- **Reward is composite** — a format reward (did
  the output parse / match the required
  structure) plus a correctness reward (did the
  answer verify). A well-formed-but-wrong answer
  and a malformed one should not score
  identically; correctness alone loses that
  signal.
- **`learning_rate=5e-7`** and **`beta=0.01`** are
  the settled starting point; deviate only after
  the base run is stable and reward-inspected
  (below).

Memory sizing for this recipe by target size
class: `references/grpo-memory.md`.

## The Inspection Rule

**Run the reward function against 50–100 sampled
outputs and manually read the results before
starting the actual training run.** This is a
gate, not a one-time sanity check.

If the reward function's judgment disagrees with
a human reading of that sample, fix the reward
function first. Training against an uninspected
reward, or tuning hyperparameters to compensate
for one silently scoring the wrong thing, is how
a run reward-hacks: the model optimizes cleanly
toward the wrong target, and that doesn't surface
as a training-loop bug.

This inspection is a Phase 1 gate input for
`/finetune` — the same 50–100-sample read that
catches a broken reward function here is what that
command checks for before it lets a GRPO brief
proceed.

Complete reward function implementations to
inspect against — exact-match, schema-validation,
unit-test-execution, a length-penalty wrapper, and
a rubric-as-reward judge pattern:
`references/reward-functions.md`.

## Variant Selection

The base recipe above is the default. Reach for a
variant only when a specific failure mode shows
up, not preemptively:

| Failure mode | Variant | Why |
|---|---|---|
| Entropy collapse / degenerate long chain-of-thought | **DAPO** | Decouples clip bounds and relaxes the KL penalty that over-regularizes exploration on long reasoning traces |
| Reward or output length trends up regardless of quality | **Dr.GRPO** | Removes GRPO's length-normalization bias so reward tracks correctness, not completion length |
| Training a mixture-of-experts model | **GSPO** | Moves the importance-sampling ratio to the sequence level instead of per-token — per-token ratios are unstable on MoE routing, so GSPO is required here, not optional |

Start with plain GRPO. Watch for the specific
symptom — collapsing entropy on long CoT, a
length-reward correlation, or MoE instability —
and only then swap in the matching variant above.
Don't pre-select a variant before the base recipe
has actually shown the failure mode.

## VLM RL Is Reference-Only

Vision-language RL is **not executed by this
plugin in v1** — it's documented here for
context, not as a runnable path. Tooling is
fragmented across ms-swift and EasyR1-derived
forks with no one-line TRL command yet, and naive
text-only GRPO applied to a VLM tends to
reward-hack by optimizing the text-reasoning trace
while ignoring the image — the model learns to
sound right without looking at the input. A VLM
RL run is a research spike outside this skill's
supported recipe, not a variant of The Recipe
above.

## References

- `references/reward-functions.md` — complete
  Python reward functions (exact-match
  correctness, schema validation, unit-test
  execution, a length-penalty wrapper, and a
  rubric-as-reward judge pattern) to inspect under
  The Inspection Rule before any training run.
- `references/grpo-memory.md` — memory sizing by
  target size class, vLLM sleep-mode and
  optimizer-state tactics, Unsloth's long-context
  RL chunking, and the DGX Spark bandwidth caveat
  for decode-heavy rollouts.

Related skills: `finetuning-method-selection`
routes here once a verifiable pass/fail signal
exists; `preference-optimization` is the sibling
skill for preference pairs rather than verifiable
rewards; `eval-harness-first` covers judge
calibration for any reward that isn't purely
code-checkable. On DGX Spark, defer to the
`dgx-spark-ops` plugin's skills, when installed,
for the memory/thermal remediation ladder this
skill's memory table doesn't cover.