Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →loulanyue/agent-eval
Head-to-head comparison of coding agents (Claude Code, Aider, Codex, etc.) on custom tasks with pass rate, cost, time, and consistency metrics
Passed 20 of 21 test cases with the skill, and 9 of 21 without it.
and 18 more in the full report
12 fixed9 passed either way1 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 11 test cases the skill fixed. Same model, same prompt, with and without the skill.
I want to compare two candidates named claude-code and aider simultaneously on task file tasks/refactor.yaml. What CLI flags do I pass to run both agents in one command?
| Checked | Without skill | With skill |
|---|---|---|
| Passes multiple --agent flags for each target agent | Missed | Passed |
| Uses --task tasks/refactor.yaml to select the task specification | Passed | Passed |
The instructions your model reads when the skill loads.
A lightweight CLI tool for comparing coding agents head-to-head on reproducible tasks. Every "which coding agent is best?" comparison runs on vibes — this tool systematizes it.
Note: Install agent-eval from its repository after reviewing the source.
Define tasks declaratively. Each task specifies what to do, which files to touch, and how to judge success:
name: add-retry-logic
description: Add exponential backoff retry to the HTTP client
repo: ./my-project
files:
- src/http_client.py
prompt: |
Add retry logic with exponential backoff to all HTTP requests.
Max 3 retries. Initial delay 1s, max delay 30s.
judge:
- type: pytest
command: pytest tests/test_http_client.py -v
- type: grep
pattern: "exponential_backoff|retry"
files: src/http_client.py
commit: "abc1234" # pin to specific commit for reproducibility
test cases passed, vs 9 / 21 without
Published skills whose descriptions closely match this one, with their headline benchmark lift.
More skills from the same source and nearby in the registry, with their headline benchmark lift.