Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →notque/agent-comparison
A/B test agent variants for quality and token cost.
Passed 19 of 22 test cases with the skill, and 5 of 22 without it.
and 16 more in the full report
16 fixed6 passed either way3 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 16 test cases the skill fixed. Same model, same prompt, with and without the skill.
Show me the bash command to inspect the top 20 lines of `agents/primary-compact.md` to verify that required frontmatter fields exist before benchmarking.
| Checked | Without skill | With skill |
|---|---|---|
| Executes head -20 agents/primary-compact.md | grep -E '^(name|description):'. | Missed | Passed |
The instructions your model reads when the skill loads.
Compare agent variants through controlled A/B benchmarks. Runs identical tasks on both agents, grades output quality with domain-specific checklists, and reports total session token cost to a working solution. This skill is exclusively for agent variant comparison — use agent-evaluation for single-agent assessment, and skill-eval for skill testing.
| Signal | Load These Files | Why |
|---|---|---|
| selecting benchmark tasks and directory layout (Phase 1) | benchmark-tasks.md | Loads detailed guidance from benchmark-tasks.md. |
| example-driven tasks, errors | examples-and-errors.md | Loads detailed guidance from examples-and-errors.md. |
| scoring solutions: 5-criteria rubric and effective cost calculation | grading-rubric.md | Loads detailed guidance from grading-rubric.md. |
| deciding when to run comparisons; December 2024 baseline data | methodology.md | Loads detailed guidance from methodology.md. |
| configuring autoresearch: targets, task formats, eval isolation modes | optimization-guide.md | Loads detailed guidance from optimization-guide.md. |
| executing Phase 5 OPTIMIZE step by step | optimize-phase.md | Loads detailed guidance from optimize-phase.md. |
| writing the Phase 4 comparison report | report-template.md | Loads detailed guidance from report-template.md. |
See
references/examples-and-errors.mdfor error handling. Seereferences/optimize-phase.mdfor Phase 5 OPTIMIZE full procedure. Seereferences/methodology.mdfor December 2024 benchmark data.
Goal: Create benchmark environment and validate both agent variants exist.
Read and follow the repository CLAUDE.md before starting any execution.
Step 1: Analyze original agent
wc -l agents/{original-agent}.md
grep "^## " agents/{original-agent}.md
grep -c '```' agents/{original-agent}.md
Step 2: Create or validate compact variant
If creating a compact variant, preserve:
test cases passed, vs 5 / 22 without
More skills from the same source and nearby in the registry, with their headline benchmark lift.