Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →Single-file deep reinforcement learning implementations (CleanRL). High-quality standalone implementations of PPO, DQN, C51, SAC, DDPG, TD3 with research-friendly features. Each algorithm is a self-contained file with ~300-500 lines. Includes Atari, MuJoCo, Procgen, PettingZoo multi-agent, and JAX variants. Use for RL algorithm reference, rapid prototyping, and understanding implementation details.
.claude/skills/mkurman-cleanrl/SKILL.md| Test case | Without → With | Effect | Δ tokens | Δ turns |
|---|---|---|---|---|
| case-02 | ✗→✓ | ▲ Improved | 2% | 0% |
| case-04 | ✗→✓ | ▲ Improved | 45% | 0% |
| case-05 | ✗→✓ | ▲ Improved | 228% | 0% |
| case-10 | ✗→✓ | ▲ Improved | 67% | 0% |
| case-15 | ✗→✓ | ▲ Improved | 130% | 0% |
-----|------|----------------| | PPO | ppo.py | python cleanrl/ppo.py --env-id CartPole-v1 | | PPO Atari | ppo_atari.py | python cleanrl/ppo_atari.py --env-id BreakoutNoFrameskip-v4 | | PPO Continuous | ppo_continuous_action.py | python cleanrl/ppo_continuous_action.py --env-id HalfCheetah-v4 | | PPO Multi-Agent | ppo_pettingzoo_ma_atari.py | python cleanrl/ppo_pettingzoo_ma_atari.py --env-id pong_v3 | | DQN | dqn.py | python cleanrl/dqn.py --env-id CartPole-v1 | | DQN Atari | dqn_atari.py | python cleanrl/dqn_atari.py --env-id BreakoutNoFrameskip-v4 | | C51 Atari | c51_atari.py | python cleanrl/c51_atari.py --env-id BreakoutNoFrameskip-v4 | | SAC Continuous | sac_continuous_action.py | python cleanrl/sac_continuous_action.py --env-id HalfCheetah-v4 | | SAC Atari | sac_atari.py | python cleanrl/sac_atari.py --env-id BreakoutNoFrameskip-v4 | | DDPG | ddpg_continuous_action.py | python cleanrl/ddpg_continuous_action.py --env-id HalfCheetah-v4 | | TD3 | td3_continuous_action.py | python cleanrl/td3_continuous_action.py --env-id HalfCheetah-v4 |
bash# Minimal PPO on CartPole python cleanrl/ppo.py \ --seed 1 \ --env-id CartPole-v1 \ --total-timesteps 50000 \ --track \ --wandb-project-name my-project # PPO on Atari (standard config) python cleanrl/ppo_atari.py \ --seed 1 \ --env-id BreakoutNoFrameskip-v4 \ --total-timesteps 10000000 \ --track \ --capture-video # PPO on MuJoCo continuous control python cleanrl/ppo_continuous_action.py \ --seed 1 \ --env-id HalfCheetah-v4 \ --total-timesteps 1000000
Key PPO Hyperparameters: | Parameter | CartPole/Classic | Atari | MuJoCo | |-----------|-----------------|-------|--------| | --total-timesteps | 50K | 10M | 1M | | --learning-rate | 2.5e-4 | 2.5e-4 | 3e-4 | | --num-envs | 4 | 8 | 1 | | --num-steps | 128 | 128 | 2048 | | --anneal-lr | True | True | False | | --gae-lambda | 0.95 | 0.95 | 0.95 | | --update-epochs | 4 | 4 | 10 | | --norm-adv | True | True | True | | --clip-coef | 0.2 | 0.1 | 0.2 | | --ent-coef | 0.01 | 0.01 | 0.0 |
bash# DQN on Atari python cleanrl/dqn_atari.py \ --seed 1 \ --env-id BreakoutNoFrameskip-v4 \ --total-timesteps 10000000 \ --buffer-size 100000 \ --learning-starts 80000 \ --target-network-frequency 1000 \ --batch-size 32 \ --track
bash# PPO on multi-agent Atari Pong python cleanrl/ppo_pettingzoo_ma_atari.py \ --seed 1 \ --env-id pong_v3 \ --total-timesteps 10000000 \ --track # Available MA environments: # pong_v3, surround_v2, tennis_v3, space_invaders_v2, # warlords_v3, combat_plane_v2, combat_tank_v2
bash# TensorBoard (runs in cleanrl/runs/) tensorboard --logdir runs # Weights & Biases (requires wandb login) python cleanrl/ppo.py --track --wandb-project-name my-project --wandb-entity my-entity # Video capture (every 100th evaluation) python cleanrl/ppo_atari.py --capture-video --env-id BreakoutNoFrameskip-v4
5-10x faster training via JAX compilation + EnvPool:
bash# Install JAX support pip install -r requirements/requirements-jax.txt # JAX PPO on Atari (ultra-fast) python cleanrl/ppo_atari_envpool_xla_jax.py \ --env-id BreakoutNoFrameskip-v4 \ --total-timesteps 10000000 # JAX DQN on Atari python cleanrl/dqn_atari_jax.py \ --env-id BreakoutNoFrameskip-v4 \ --total-timesteps 10000000
bash# Build Docker image docker build -t cleanrl . # Submit to AWS Batch python cleanrl/ppo_atari.py \ --env-id BreakoutNoFrameskip-v4 \ --total-timesteps 10000000 \ --track \ --upload-model
Each file follows a consistent structure:
python# 1. Imports # 2. parse_args() — CLI arguments # 3. make_env() — Environment creation # 4. Agent class (if needed) — Neural network, usually simple MLP/CNN # 5. main(): # a. Setup: seeding, device, envs # b. Initialize agent, optimizer # c. Initialize storage (rollout buffer, replay buffer) # d. Training loop: # - Collect experience # - Compute returns/advantages # - Update policy/value/Q-network # - Log metrics # e. Save model, upload
Each file is ~300-500 lines and is meant to be read top-to-bottom.
bash# Minimal test run (fewer steps, more frequent logging) python cleanrl/ppo.py \ --env-id CartPole-v1 \ --total-timesteps 5000 \ --num-envs 1 \ --num-steps 32 \ --track # Disable wandb (pure TensorBoard) python cleanrl/ppo.py --env-id CartPole-v1 --total-timesteps 50000 # Check available env IDs python -c "import gymnasium as gym; print([e for e in gym.envs.registry if 'CartPole' in e])"
import cleanrl, run the scripts directlyppo.py and modify it for your research--track for W&B logging, omit for plain TensorBoard--capture-video saves agent gameplay — great for qualitative evaluationjax.lax.scan| Case | Status | Duration (ms) | Turns | Tokens | Tool calls | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Without | With | Δ | Without | With | Δ | Without | With | Δ | Without | With | Δ | ||
case-06 | pass→pass | 11,682 | 9,090 | -22% | 1 | 1 | 0% | 2,017 | 3,699 | +83% | 0 | 0 | — |
case-01 | fail→fail | 7,645 | 4,926 | -36% | 1 | 1 | 0% | 1,692 | 3,045 | +80% | 0 | 0 | — |
case-02 | fail→pass | 15,031 | 3,418 | -77% | 1 | 1 | 0% | 2,746 | 2,811 | +2% | 0 | 0 | — |
case-03 | pass→pass | 10,909 | 3,258 | -70% | 1 | 1 | 0% | 2,007 | 2,671 | +33% | 0 | 0 | — |
case-04 | fail→pass | 15,850 | 10,386 | -34% | 1 | 1 | 0% | 2,983 | 4,335 | +45% | 0 | 0 | — |
case-05 | fail→pass | 6,893 | 10,018 | +45% | 1 | 1 | 0% | 1,160 | 3,805 | +228% | 0 | 0 | — |
case-07 | fail→fail | 4,078 | 2,858 | -30% | 1 | 1 | 0% | 705 | 2,662 | +278% | 0 | 0 | — |
case-08 | pass→pass | 9,258 | 2,549 | -72% | 1 | 1 | 0% | 1,738 | 2,575 | +48% | 0 | 0 | — |
case-09 | pass→pass | 9,656 | 2,678 | -72% | 1 | 1 | 0% | 1,586 | 2,587 | +63% | 0 | 0 | — |
case-10 | fail→pass | 8,184 | 1,444 | -82% | 1 | 1 | 0% | 1,399 | 2,335 | +67% | 0 | 0 | — |
case-11 | pass→pass | 3,212 | 1,549 | -52% | 1 | 1 | 0% | 534 | 2,398 | +349% | 0 | 0 | — |
case-12 | pass→pass | 6,812 | 3,206 | -53% | 1 | 1 | 0% | 1,172 | 2,586 | +121% | 0 | 0 | — |
case-13 | pass→pass | 8,807 | 3,242 | -63% | 1 | 1 | 0% | 1,596 | 2,730 | +71% | 0 | 0 | — |
case-14 | pass→pass | 12,297 | 2,815 | -77% | 1 | 1 | 0% | 2,301 | 2,697 | +17% | 0 | 0 | — |
case-15 | fail→pass | 6,541 | 2,320 | -65% | 1 | 1 | 0% | 1,082 | 2,492 | +130% | 0 | 0 | — |
case-16 | pass→pass | 7,125 | 1,862 | -74% | 1 | 1 | 0% | 1,193 | 2,485 | +108% | 0 | 0 | — |
case-17 | pass→pass | 7,212 | 3,301 | -54% | 1 | 1 | 0% | 1,354 | 2,719 | +101% | 0 | 0 | — |
case-18 | pass→pass | 4,691 | 2,667 | -43% | 1 | 1 | 0% | 757 | 2,611 | +245% | 0 | 0 | — |
case-19 | pass→pass | 10,558 | 2,769 | -74% | 1 | 1 | 0% | 1,790 | 2,643 | +48% | 0 | 0 | — |
case-20 | fail→pass | 4,280 | 1,965 | -54% | 1 | 1 | 0% | 761 | 2,484 | +226% | 0 | 0 | — |
case-21 | pass→pass | 2,433 | 2,172 | -11% | 1 | 1 | 0% | 449 | 2,519 | +461% | 0 | 0 | — |
case-22 | pass→pass | 9,470 | 7,485 | -21% | 1 | 1 | 0% | 1,561 | 3,309 | +112% | 0 | 0 | — |
case-23 | pass→pass | 9,654 | 8,249 | -15% | 1 | 1 | 0% | 1,617 | 3,098 | +92% | 0 | 0 | — |
DecimalAI ran this skill against gemini-3.6-flash twice over the same eval suite — once with the skill loaded and once without — and compared the two runs case by case. 23 cases were attempted. The headline lift of +26 percentage points is the difference between those two pass rates over the 23 comparable cases. 1 case got worse with the skill loaded, and it is included in that figure.
Without the skill loaded, the model failed this case. With it loaded, the same prompt on the same model passed. This is one improved case from the latest verified run; every case, including any that regressed, is in the table above.
Other measured skills in the registry, with their headline benchmark lift.