Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →Compare Claude Code and Codex on the same real code-change task with isolated worktrees, identical gates, transcripts, time, and cost.
.claude/skills/hashgraph-online-ai-agent-bench/SKILL.md| Test case | Without → With | Effect | Δ tokens | Δ turns |
|---|---|---|---|---|
| case-07 | ✗→✓ | ▲ Improved | -60% | 0% |
| case-04 | ✗→✓ | ▲ Improved | -59% | 0% |
| case-05 | ✗→✓ | ▲ Improved | -18% | 0% |
| case-06 | ✗→✓ | ▲ Improved | -52% | 0% |
| case-08 | ✗→✓ | ▲ Improved | -27% | 0% |
Compare agents only with the same task, starting commit, and outcome check. The harness preserves result branches and removes temporary worktrees.
Create <repo>/.agent-bench.toml:
tomlprompt = "prompts/task.md" start_branch = "main" # or start_commit agents = ["claude", "codex"] outer_check = "./scripts/full_check.sh" inner_check = "pytest tests/integration/test_x.py -q"
outer_check proves the real outcome before and after, and measures wall time. inner_check gives agents fast feedback.
Require a clean repo, available CLIs, and a passing outer_check. Confirm agents and run ID, then run trials sequentially to avoid load-biased timing:
bashpython <skill>/scripts/run_trial.py --repo "$REPO" --config "$REPO/.agent-bench.toml" --agent "$AGENT" --run "$RUN_ID"
Results go to eval-results/<task>/<agent>/run-<id>-<timestamp>/. Record unexpected behavior in ai-agent-bench-anomalies.md per anomalies.
Aggregate with scripts/parse_transcript.py --aggregate <run-dirs> --output comparison.json --render-report comparison.md. Report gates, branches, time delta, tokens, and cost. Never rank a failed trial.
For plugin behavior rather than a real code task, use the bounded Pydantic runner documented by eval-regression and scripts/run_evals.py.
Never commit on the user's branch. A repeated run creates a new timestamped result and preserves prior evidence.
| Case | Status | Duration (ms) | Turns | Tokens | Tool calls | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Without | With | Δ | Without | With | Δ | Without | With | Δ | Without | With | Δ | ||
case-07 | fail→pass | 17,311 | 6,118 | -65% | 1 | 1 | 0% | 2,138 | 855 | -60% | 0 | 0 | — |
case-01 | fail→fail | 45,406 | 8,659 | -81% | 1 | 1 | 0% | 8,906 | 565 | -94% | 0 | 0 | — |
case-02 | fail→fail | 35,329 | 5,979 | -83% | 1 | 1 | 0% | 8,230 | 603 | -93% | 0 | 0 | — |
case-03 | fail→fail | 8,266 | 14,693 | +78% | 1 | 1 | 0% | 194 | 569 | +193% | 0 | 0 | — |
case-04 | fail→pass | 24,689 | 4,357 | -82% | 1 | 1 | 0% | 3,092 | 1,262 | -59% | 0 | 0 | — |
case-05 | fail→pass | 14,106 | 14,327 | +2% | 1 | 1 | 0% | 2,625 | 2,161 | -18% | 0 | 0 | — |
case-06 | fail→pass | 9,438 | 7,648 | -19% | 1 | 1 | 0% | 1,735 | 838 | -52% | 0 | 0 | — |
case-08 | fail→pass | 13,700 | 3,465 | -75% | 1 | 1 | 0% | 1,308 | 956 | -27% | 0 | 0 | — |
case-09 | fail→pass | 14,113 | 7,239 | -49% | 1 | 1 | 0% | 2,302 | 720 | -69% | 0 | 0 | — |
case-10 | pass→pass | 16,327 | 2,177 | -87% | 1 | 1 | 0% | 1,734 | 723 | -58% | 0 | 0 | — |
case-11 | fail→pass | 10,846 | 3,788 | -65% | 1 | 1 | 0% | 1,800 | 1,036 | -42% | 0 | 0 | — |
case-12 | fail→pass | 27,557 | 6,583 | -76% | 1 | 1 | 0% | 3,992 | 604 | -85% | 0 | 0 | — |
case-13 | fail→pass | 8,467 | 2,187 | -74% | 1 | 1 | 0% | 1,263 | 728 | -42% | 0 | 0 | — |
case-14 | fail→fail | 11,357 | 1,719 | -85% | 1 | 1 | 0% | 1,982 | 652 | -67% | 0 | 0 | — |
case-15 | fail→pass | 9,384 | 7,190 | -23% | 1 | 1 | 0% | 1,514 | 749 | -51% | 0 | 0 | — |
case-16 | fail→pass | 18,628 | 10,606 | -43% | 1 | 1 | 0% | 2,663 | 1,340 | -50% | 0 | 0 | — |
case-17 | fail→pass | 13,862 | 3,604 | -74% | 1 | 1 | 0% | 2,288 | 1,049 | -54% | 0 | 0 | — |
case-18 | fail→pass | 14,748 | 1,795 | -88% | 1 | 1 | 0% | 1,642 | 698 | -57% | 0 | 0 | — |
case-19 | pass→pass | 11,888 | 13,094 | +10% | 1 | 1 | 0% | 2,470 | 1,988 | -20% | 0 | 0 | — |
case-20 | pass→pass | 6,165 | 10,104 | +64% | 1 | 1 | 0% | 1,208 | 1,316 | +9% | 0 | 0 | — |
case-21 | pass→pass | 16,557 | 11,165 | -33% | 1 | 1 | 0% | 1,830 | 1,596 | -13% | 0 | 0 | — |
case-22 | pass→pass | 18,365 | 3,851 | -79% | 1 | 1 | 0% | 2,232 | 974 | -56% | 0 | 0 | — |
DecimalAI ran this skill against gemini-3.6-flash twice over the same eval suite — once with the skill loaded and once without — and compared the two runs case by case. 22 cases were attempted, and 19 counted toward the lift figure. The other 3 produced results that are not comparable between the two arms, so they are excluded from the headline rather than averaged into it. The headline lift of +59 percentage points is the difference between those two pass rates over the 19 comparable cases.
Without the skill loaded, the model failed this case. With it loaded, the same prompt on the same model passed. This is one improved case from the latest verified run; every case, including any that regressed, is in the table above.
Other measured skills in the registry, with their headline benchmark lift.