Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

Agent Benchmark Framework · vibeeval | DecimalAI

ai mlby vibeeval

Agent Benchmark Framework

agent-benchmark

Source vibeeval/vibecosystem ↗License MIT

Framework for measuring and tracking agent response quality over time. Detects regressions before they reach production.

SkillScore77Maintained

Automatic

AI Engineer pack

Last verified 7/31/2026

What it changes

Passed 16 of 21 test cases with the skill, and 3 of 21 without it.

Without skill14%

With skill77%

Checks it passed that the model alone missed

Categorizes the status as WARN
Awards 10 points for missing 1 critical issue
Awards 22 points for missing 1 non-critical issue

and 16 more in the full report

14 fixed3 passed either way5 still failing0 broke

Pass-rate lift: +64 pp
Tokens: +121%
Agent turns: 0%
Response time: -65%

Model gemini-3.6-flashMeasured 7/31/2026v117 of 22 cases passed with skill · 21 comparable

Tokens, turns and time describe resource use, not a dollar price.

Use this skill

Save it into your project's skills folder. Claude Code picks it up in its next session.

curl -fsSL --create-dirs -o .claude/skills/agent-benchmark/SKILL.md https://app.decimal.ai/s/agent-benchmark/SKILL.md

Want it pinned and updatable? pip install "decimalai>=0.15", then decimalai skills add agent-benchmark, and start Claude Code with claude --plugin-dir "$(git rev-parse --show-toplevel)/.agents"

1 · One-time — install the CLI (docs ↗)

pip install "decimalai>=0.15"

2 · Get the skill — writes .agents/skills/agent-benchmark/ and pins it in .decimal/skills.lock

decimalai skills add agent-benchmark

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/agent-benchmark/SKILL.md https://app.decimal.ai/s/agent-benchmark/SKILL.md

Raw file: /s/agent-benchmark/SKILL.md ↗ · machine-readable summary at /s/agent-benchmark.json ↗

Safety checks: 3 of 3 passed

Code scan CleanIntent PassedContent Clean

See it in action

One of 13 test cases the skill fixed. Same model, same prompt, with and without the skill.

1 of 13

Prompt

When setting up benchmark fixtures for the verifier agent, how many total fixtures are defined and what critical requirement is scored over prose quality?

What was checked, without and with the skill
Checked	Without skill	With skill
Defines a total of 4 fixtures for the verifier agent	Missed	Passed
Prioritizes verdict accuracy over prose quality	Passed	Passed

case-13 · gemini-3.6-flash · 7/31/2026 · answers are capped and redacted

What's inside

The instructions your model reads when the skill loads.

SKILL.md407 linessingle file

Agent Benchmark Framework

Without benchmarks, we cannot know whether agent changes improve or degrade quality. This skill defines how to measure, track, and protect agent performance.

When to Activate

Before and after modifying any agent definition file
When adding a new skill that an agent depends on
Periodic quality audits (weekly/monthly)
When a user reports degraded agent output
Before promoting an agent from experimental to production

Core Concepts

Why Benchmarks Matter

Agent quality degrades silently. A prompt tweak that improves one response can break ten others. Without a baseline to compare against, every change is a guess. Benchmarks make quality visible and regressions detectable.

Benchmark Types

Type	Scope	Cost	Frequency
Prompt Benchmark	Single agent, single task	Low	Every agent change
Task Benchmark	End-to-end scenario	Medium	Feature changes
Regression Suite	All critical agents	High	Weekly / before release

Directory Structure

~/.claude/benchmarks/
  fixtures/
    code-reviewer/
      missing-error-handling.ts      # Input: code with no try/catch
      sql-injection.py               # Input: unparameterized query
      clean-code.ts                  # Input: code with no issues
    security-reviewer/
      hardcoded-secret.ts            # Input: API key in source
      parameterized-query.py         # Input: safe query (no findings expected)
    verifier/
      passing-build/                 # Input: project that builds

Track record

Benchmark

16 / 21

test cases passed, vs 3 / 21 without

Adoption

Users: 0
Times used: 0

Safety and source details

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v4 — 7/30/2026

View 15 scanner checks

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

vibeeval / vibecosystem

skills/agent-benchmark/SKILL.md

View on GitHub ↗

License MITImported 8/1/2026Version v1

More skills from the same source and nearby in the registry, with their headline benchmark lift.

Ready to supercharge your AI agents?

See it in action

What's inside

Agent Benchmark Framework

When to Activate

Core Concepts

Why Benchmarks Matter

Benchmark Types

Directory Structure

Track record

Benchmark

Adoption

SkillSafety

Source

Related skills