Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

Minimal PPO on CartPole · mkurman | DecimalAI

ai mlby mkurman

Minimal PPO on CartPole

mkurman/cleanrl

Source mkurman/zorai ↗License MIT

Single-file deep reinforcement learning implementations (CleanRL). High-quality standalone implementations of PPO, DQN, C51, SAC, DDPG, TD3 with research-friendly features. Each algorithm is a self-contained file with ~300-500 lines. Includes Atari, MuJoCo, Procgen, PettingZoo multi-agent, and JAX variants.

SkillScore91Maintained

Automatic

AI Engineer pack

Last verified 8/21/2026

What it changes

Passed 21 of 23 test cases with the skill, and 15 of 23 without it.

Without skill65%

With skill91%

Checks it passed that the model alone missed

The command sets buffer size to 100000.
The command sets target network frequency to 1000.
Identifies the script path as cleanrl/c51_atari.py.

and 6 more in the full report

6 fixed15 passed either way2 still failing0 broke

Pass-rate lift: +26 pp
Tokens: +91%
Agent turns: 0%
Response time: -51%

Model gemini-3.6-flashMeasured 8/21/2026v121 of 23 cases passed with skill

Tokens, turns and time describe resource use, not a dollar price.

Use this skill

Save it into your project's skills folder. Claude Code picks it up in its next session.

curl -fsSL --create-dirs -o .claude/skills/mkurman-cleanrl/SKILL.md https://app.decimal.ai/s/mkurman-cleanrl/SKILL.md

Want it pinned and updatable? pip install "decimalai>=0.15", then decimalai skills add mkurman-cleanrl, and start Claude Code with claude --plugin-dir "$(git rev-parse --show-toplevel)/.agents"

1 · One-time — install the CLI (docs ↗)

pip install "decimalai>=0.15"

2 · Get the skill — writes .agents/skills/mkurman-cleanrl/ and pins it in .decimal/skills.lock

decimalai skills add mkurman-cleanrl

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/mkurman-cleanrl/SKILL.md https://app.decimal.ai/s/mkurman-cleanrl/SKILL.md

Raw file: /s/mkurman-cleanrl/SKILL.md ↗ · machine-readable summary at /s/mkurman-cleanrl.json ↗

Safety checks: 3 of 3 passed

Code scan CleanIntent PassedContent Clean

See it in action

One of 6 test cases the skill fixed. Same model, same prompt, with and without the skill.

1 of 6

Prompt

I am setting up multi-agent RL benchmarks with the PettingZoo Atari script. Beyond pong_v3, name three other supported PettingZoo environment IDs.

What was checked, without and with the skill
Checked	Without skill	With skill
Includes surround_v2 among the listed environments.	Missed	Passed
Includes tennis_v3 among the listed environments.	Passed	Passed
Includes space_invaders_v2 among the listed environments.	Missed	Passed

case-20 · gemini-3.6-flash · 8/21/2026 · answers are capped and redacted

What's inside

The instructions your model reads when the skill loads.

SKILL.md184 linessingle file

-----|------|----------------| | PPO | ppo.py | python cleanrl/ppo.py --env-id CartPole-v1 | | PPO Atari | ppo_atari.py | python cleanrl/ppo_atari.py --env-id BreakoutNoFrameskip-v4 | | PPO Continuous | ppo_continuous_action.py | python cleanrl/ppo_continuous_action.py --env-id HalfCheetah-v4 | | PPO Multi-Agent | ppo_pettingzoo_ma_atari.py | python cleanrl/ppo_pettingzoo_ma_atari.py --env-id pong_v3 | | DQN | dqn.py | python cleanrl/dqn.py --env-id CartPole-v1 | | DQN Atari | dqn_atari.py | python cleanrl/dqn_atari.py --env-id BreakoutNoFrameskip-v4 | | C51 Atari | c51_atari.py | python cleanrl/c51_atari.py --env-id BreakoutNoFrameskip-v4 | | SAC Continuous | sac_continuous_action.py | python cleanrl/sac_continuous_action.py --env-id HalfCheetah-v4 | | SAC Atari | sac_atari.py | python cleanrl/sac_atari.py --env-id BreakoutNoFrameskip-v4 | | DDPG | ddpg_continuous_action.py | python cleanrl/ddpg_continuous_action.py --env-id HalfCheetah-v4 | | TD3 | td3_continuous_action.py | python cleanrl/td3_continuous_action.py --env-id HalfCheetah-v4 |

3. PPO Training Workflow

# Minimal PPO on CartPole
python cleanrl/ppo.py \
    --seed 1 \
    --env-id CartPole-v1 \
    --total-timesteps 50000 \
    --track \
    --wandb-project-name my-project

# PPO on Atari (standard config)
python cleanrl/ppo_atari.py \
    --seed 1 \
    --env-id BreakoutNoFrameskip-v4 \
    --total-timesteps 10000000 \
    --track \
    --capture-video

# PPO on MuJoCo continuous control
python cleanrl/ppo_continuous_action.py \
    --seed 1 \
    --env-id HalfCheetah-v4 \
    --total-timesteps 1000000

Key PPO Hyperparameters:

Track record

Benchmark

21 / 23

test cases passed, vs 15 / 23 without

Adoption

Users: 0
Times used: 0

Safety and source details

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v4 — 8/3/2026

View 15 scanner checks

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

mkurman / zorai

skills/scientific-skills/cleanrl/SKILL.md

View on GitHub ↗

License MIT licenseImported 8/3/2026Version v1

More skills from the same source and nearby in the registry, with their headline benchmark lift.