Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →nousresearch/fine-tuning-with-trl
TRL: SFT, DPO, GRPO, RLOO reward modeling for LLM RLHF.
Passed 21 of 22 test cases with the skill, and 19 of 22 without it.
2 fixed20 passed either way1 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 2 test cases the skill fixed. Same model, same prompt, with and without the skill.
I've finished fine-tuning my base model and training a sequence classification reward model for my dataset. Now I want to perform online reinforcement learning using that reward model to optimize the policy in TRL. Can you share a Python script or CLI command structure showing how to configure and run this reward-guided online RL training stage?
| Checked | Without skill | With skill |
|---|---|---|
| Does not recommend PPOTrainer. | Missed | Passed |
| Recommends RLOOTrainer for online reinforcement learning. | Missed | Passed |
The instructions your model reads when the skill loads.
TRL provides post-training methods for aligning language models with human preferences.
Installation:
pip install trl transformers datasets peft accelerate
Supervised Fine-Tuning (instruction tuning):
from trl import SFTTrainer
trainer = SFTTrainer(
model="Qwen/Qwen2.5-0.5B",
train_dataset=dataset, # Prompt-completion pairs
)
trainer.train()
DPO (align with preferences):
from trl import DPOTrainer, DPOConfig
config = DPOConfig(output_dir="model-dpo", beta=0.1)
trainer = DPOTrainer(
model=model,
args=config,
train_dataset=preference_dataset, # chosen/rejected pairs
processing_class=tokenizer
)
trainer.train()
test cases passed, vs 19 / 22 without
Published skills whose descriptions closely match this one, with their headline benchmark lift.
More skills from the same source and nearby in the registry, with their headline benchmark lift.