Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

Auto Arena Skill · agentscope-ai | DecimalAI

ai mlby agentscope-ai

Auto Arena Skill

agentscope-ai/01-auto-arena

Source agentscope-ai/openjudge ↗License Apache-2.0

Automatically evaluate and compare multiple AI models or agents without pre-existing test data. Generates test queries from a task description, collects responses from all target endpoints, auto-generates evaluation rubrics, runs pairwise comparisons via a judge model, and produces win-rate rankings with reports and charts. Supports checkpoint resume, incremental endpoint addition, and judge model hot-swap.

SkillScore100Maintained

Automatic

AI Engineer pack

Last verified 10/11/2026

What it changes

Passed 22 of 22 test cases with the skill, and 9 of 22 without it.

Without skill41%

With skill100%

Checks it passed that the model alone missed

Specifies base_url for the custom endpoint
Executes evaluation using pipeline.evaluate
Includes the --save flag to save output results

and 19 more in the full report

13 fixed9 passed either way0 still failing0 broke1 errored

Pass-rate lift: +59 pp
Tokens: +62%
Agent turns: 0%
Response time: -51%

Model gemini-3.6-flashMeasured 10/11/2026v122 of 23 cases passed with skill · 22 comparable

Tokens, turns and time describe resource use, not a dollar price.

Use this skill

Save it into your project's skills folder. Claude Code picks it up in its next session.

curl -fsSL --create-dirs -o .claude/skills/agentscope-ai-01-auto-arena/SKILL.md https://app.decimal.ai/s/agentscope-ai-01-auto-arena/SKILL.md

Want it pinned and updatable? pip install "decimalai>=0.15", then decimalai skills add agentscope-ai-01-auto-arena, and start Claude Code with claude --plugin-dir "$(git rev-parse --show-toplevel)/.agents"

1 · One-time — install the CLI (docs ↗)

pip install "decimalai>=0.15"

2 · Get the skill — writes .agents/skills/agentscope-ai-01-auto-arena/ and pins it in .decimal/skills.lock

decimalai skills add agentscope-ai-01-auto-arena

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/agentscope-ai-01-auto-arena/SKILL.md https://app.decimal.ai/s/agentscope-ai-01-auto-arena/SKILL.md

Raw file: /s/agentscope-ai-01-auto-arena/SKILL.md ↗ · machine-readable summary at /s/agentscope-ai-01-auto-arena.json ↗

Safety checks: 3 of 3 passed

Code scan CleanIntent PassedContent Clean

See it in action

One of 13 test cases the skill fixed. Same model, same prompt, with and without the skill.

1 of 13

Prompt

I am enabling the Markdown evaluation report in my arena configuration and want to include exactly 5 illustrative example comparisons per section instead of the default 3. How should this be specified in `config.yaml`?

What was checked, without and with the skill
Checked	Without skill	With skill
Configures include_examples set to 5 under the report section	Missed	Passed

case-20 · gemini-3.6-flash · 10/11/2026 · answers are capped and redacted

What's inside

The instructions your model reads when the skill loads.

SKILL.md261 linessingle file

Auto Arena Skill

End-to-end automated model comparison using the OpenJudge AutoArenaPipeline:

Generate queries — LLM creates diverse test queries from task description
Collect responses — query all target endpoints concurrently
Generate rubrics — LLM produces evaluation criteria from task + sample queries
Pairwise evaluation — judge model compares every model pair (with position-bias swap)
Analyze & rank — compute win rates, win matrix, and rankings
Report & charts — Markdown report + win-rate bar chart + optional matrix heatmap

Prerequisites

# Install OpenJudge
pip install py-openjudge

# Extra dependency for auto_arena (chart generation)
pip install matplotlib

Gather from user before running

Info	Required?	Notes
Task description	Yes	What the models/agents should do (set in config YAML)
Target endpoints	Yes	At least 2 OpenAI-compatible endpoints to compare
Judge endpoint	Yes	Strong model for pairwise evaluation (e.g. `gpt-4`, `qwen-max`)
API keys	Yes	Env vars: `OPENAI_API_KEY`, `DASHSCOPE_API_KEY`, etc.
Number of queries	No	Default: `20`
Seed queries	No	Example queries to guide generation style
System prompts	No	Per-endpoint system prompts
Output directory	No	Default: `./evaluation_results`
Report language	No	`"zh"` (default) or `"en"`

Quick start

CLI

Track record

Benchmark

22 / 22

test cases passed, vs 9 / 22 without

Adoption

Users: 0
Times used: 0

Safety and source details

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v5 — 9/17/2026

View 15 scanner checks

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

agentscope-ai / openjudge

skills/arena-eval/01-auto-arena/SKILL.md

View on GitHub ↗

License Apache-2.0Imported 9/17/2026Version v1

Other skills for the same job

Published skills whose descriptions closely match this one, with their headline benchmark lift.

agentscope-ai/auto-arenaAutomatically evaluate and compare multiple AI models or agents without pre-existing test data. Generates test queries from a task description, collects responses from all target endpoints, auto-generates evaluation rubrics, runs pairwise comparisons via a judge model, and produces win-rate rankings with reports and charts. Supports checkpoint resume, incremental endpoint addition, and judge model hot-swap. Use when the user asks to compare, benchmark, or rank multiple models or agents on a custNear-identical description+55 pp pass-rate lift

More skills from the same source and nearby in the registry, with their headline benchmark lift.

Ready to supercharge your AI agents?

See it in action

What's inside

Auto Arena Skill

Prerequisites

Gather from user before running

Quick start

CLI

Track record

Benchmark

Adoption

SkillSafety

Source

Other skills for the same job

Related skills