Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

Eval Design · agentscope-ai | DecimalAI

ai mlby agentscope-ai

Eval Design

agentscope-ai/eval-design

Source agentscope-ai/openjudge ↗License Apache-2.0

Use when the user needs to design evaluation datasets, create test cases, stratify samples, generate adversarial examples, extract eval dimensions from traces/specs, or build a labeled evaluation set. Also use when the user mentions test data design, eval coverage, difficulty stratification, synthetic data generation for eval, or "how to create good evaluation data." Outputs datasets in OpenJudge-compatible format.

SkillScore86Maintained

Automatic

AI Engineer pack

Last verified 8/14/2026

What it changes

Passed 19 of 22 test cases with the skill, and 12 of 22 without it.

Without skill55%

With skill86%

Checks it passed that the model alone missed

Passes the --self-test flag to coverage_check.py
Executes scripts/coverage_check.py with the --dataset flag
Flags coverage as thin when a dimension has fewer than 5 samples

and 4 more in the full report

7 fixed12 passed either way3 still failing0 broke

Pass-rate lift: +32 pp
Tokens: +71%
Agent turns: 0%
Response time: -36%

Model gemini-3.6-flashMeasured 8/14/2026v119 of 22 cases passed with skill

Tokens, turns and time describe resource use, not a dollar price.

Use this skill

Save it into your project's skills folder. Claude Code picks it up in its next session.

curl -fsSL --create-dirs -o .claude/skills/agentscope-ai-eval-design/SKILL.md https://app.decimal.ai/s/agentscope-ai-eval-design/SKILL.md

Want it pinned and updatable? pip install "decimalai>=0.15", then decimalai skills add agentscope-ai-eval-design, and start Claude Code with claude --plugin-dir "$(git rev-parse --show-toplevel)/.agents"

1 · One-time — install the CLI (docs ↗)

pip install "decimalai>=0.15"

2 · Get the skill — writes .agents/skills/agentscope-ai-eval-design/ and pins it in .decimal/skills.lock

decimalai skills add agentscope-ai-eval-design

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/agentscope-ai-eval-design/SKILL.md https://app.decimal.ai/s/agentscope-ai-eval-design/SKILL.md

Raw file: /s/agentscope-ai-eval-design/SKILL.md ↗ · machine-readable summary at /s/agentscope-ai-eval-design.json ↗

Safety checks: 3 of 3 passed

Code scan CleanIntent PassedContent Clean

See it in action

One of 7 test cases the skill fixed. Same model, same prompt, with and without the skill.

1 of 7

Prompt

What script name and command line invocation should be used to validate evaluation dataset coverage locally without relying on external OpenJudge runtime dependencies?

What was checked, without and with the skill
Checked	Without skill	With skill
Executes scripts/coverage_check.py with the --dataset flag	Missed	Passed

case-08 · gemini-3.6-flash · 8/14/2026 · answers are capped and redacted

What's inside

The instructions your model reads when the skill loads.

SKILL.md254 lines2 filesincludes scripts

Eval Design

Design high-quality evaluation datasets that measure what actually matters for your application. You extract evaluation dimensions from business context, structure them into stratified test cases, and output datasets ready for OpenJudge GradingRunner.

When to Activate

User has agent traces / production logs and wants to build an eval set from them
User has evaluation principles but needs properly stratified test data
User wants to generate adversarial examples that stress-test their system
User needs coverage analysis — are they testing all the right things?
User wants a labeling guide for human annotators

Checklist

You MUST create a task for each item and complete them in order:

Extract eval dimensions — from traces, spec, or user interview
Design stratified sampling — 60/30/10 split with difficulty strata
Generate test data — synthetic inputs + adversarial examples
Output OpenJudge dataset — structured format ready for GradingRunner

Coverage check: run the bundled script

After you have a dataset, validate coverage with the bundled, tested script (scripts/coverage_check.py, standard library only, no OpenJudge dependency) before trusting any per-slice metric:

python scripts/coverage_check.py --dataset eval-data/dataset.jsonl

It reports per-dimension and per-(dimension × stratum) counts, flags thin cells (< 5 per dimension, < 10 per cell), checks the adversarial share (≥ 10%), and returns a verdict (adequate / thin_coverage; exit 0 if adequate). --self-test to verify it.

Step 1: Extract Evaluation Dimensions

From traces (when user has production data)

Track record

Benchmark

19 / 22

test cases passed, vs 12 / 22 without

Adoption

Users: 0
Times used: 0

Safety and source details

SkillSafety

✓ 3 of 3 passed

Code scan14 of 15 checks clean

0 critical · 0 warning · 1 info — scanner v4 — 8/3/2026

View 15 scanner checks

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

No issues (informational notes only).

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

agentscope-ai / openjudge

skills/eval_pipeline/01-eval-design/SKILL.md

View on GitHub ↗

License Apache-2.0Imported 8/3/2026Version v1Bundled 1 file · scripts

More skills from the same source and nearby in the registry, with their headline benchmark lift.

Ready to supercharge your AI agents?

See it in action

What's inside

Eval Design

When to Activate

Checklist

Coverage check: run the bundled script

Step 1: Extract Evaluation Dimensions

From traces (when user has production data)

Track record

Benchmark

Adoption

SkillSafety

Source

Related skills