Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

Get Started Free →

Auditing an Eval Suite · growthxai | DecimalAI

ai mlby growthxai

Auditing an Eval Suite

growthxai/output-eval-audit

Source growthxai/output ↗License Apache-2.0

Audit an existing eval suite for trustworthiness.

SkillScore91Maintained

Automatic

AI Engineer pack

Last verified 8/20/2026

What it changes

Passed 17 of 19 test cases with the skill, and 12 of 19 without it.

Without skill64%

With skill91%

Checks it passed that the model alone missed

Evaluates all four standard eval diagnostic areas.
Assigns highest priority ranking to error analysis.
Assigns judge prompt splitting as priority 2 in remediation order.

and 8 more in the full report

6 fixed14 passed either way2 still failing0 broke

Pass-rate lift: +27 pp
Tokens: +75%
Agent turns: 0%
Response time: -37%

Model gemini-3.6-flashMeasured 8/20/2026v120 of 22 cases passed with skill · 19 comparable

Tokens, turns and time describe resource use, not a dollar price.

Use this skill

Save it into your project's skills folder. Claude Code picks it up in its next session.

curl -fsSL --create-dirs -o .claude/skills/growthxai-output-eval-audit/SKILL.md https://app.decimal.ai/s/growthxai-output-eval-audit/SKILL.md

Want it pinned and updatable? pip install "decimalai>=0.15", then decimalai skills add growthxai-output-eval-audit, and start Claude Code with claude --plugin-dir "$(git rev-parse --show-toplevel)/.agents"

1 · One-time — install the CLI (docs ↗)

pip install "decimalai>=0.15"

2 · Get the skill — writes .agents/skills/growthxai-output-eval-audit/ and pins it in .decimal/skills.lock

decimalai skills add growthxai-output-eval-audit

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/growthxai-output-eval-audit/SKILL.md https://app.decimal.ai/s/growthxai-output-eval-audit/SKILL.md

Raw file: /s/growthxai-output-eval-audit/SKILL.md ↗ · machine-readable summary at /s/growthxai-output-eval-audit.json ↗

Safety checks: 3 of 3 passed

Code scan CleanIntent PassedContent Clean

See it in action

One of 4 test cases the skill fixed. Same model, same prompt, with and without the skill.

1 of 4

Prompt

When validating our LLM judges for `finance-analyzer` against human labels, what are the minimum acceptable thresholds for TPR and TNR, and what is the recommended target threshold?

What was checked, without and with the skill
Checked	Without skill	With skill
States minimum acceptable threshold for TPR and TNR as 80 percent.	Passed	Passed
States target threshold for TPR and TNR as greater than 90 percent.	Missed	Passed

case-22 · gemini-3.6-flash · 8/20/2026 · answers are capped and redacted

What's inside

The instructions your model reads when the skill loads.

SKILL.md233 linessingle file

Auditing an Eval Suite

Overview

Audit your eval suite to determine whether it actually catches real failures. This skill provides a structured diagnostic that identifies gaps in error analysis, evaluator design, judge validation, and dataset coverage, with concrete remediation steps for each finding.

When to Use

Inheriting an eval suite from another team or developer
Suspecting that evals pass but production quality is poor
After switching models, rewriting prompts, or changing pipeline logic
Periodic health check (quarterly or after major releases)

Step 1: Gather Artifacts

Read the eval infrastructure files for the workflow being audited:

src/workflows/<workflow_name>/
├── tests/
│   ├── datasets/           # YAML dataset files
│   │   ├── *.yml
│   │   └── ...
│   └── evals/
│       ├── evaluators.ts   # Evaluator definitions
│       ├── workflow.ts      # Eval workflow definition
│       └── *.prompt         # Judge prompt files

Inventory what exists:

Artifact	File(s)	Count
Evaluators	`tests/evals/evaluators.ts`	?
Eval workflow	`tests/evals/workflow.ts`	? entries in `evals` array
Judge prompts	`tests/evals/*.prompt`	?
Datasets	`tests/datasets/*.yml`	?
Datasets with ground_truth	? of above	?
Datasets with last_output	? of above	?

Track record

Benchmark

17 / 19

test cases passed, vs 12 / 19 without

Adoption

Users: 0
Times used: 0

Safety and source details

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v4 — 8/3/2026

View 15 scanner checks

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

growthxai / output

coding_assistants/claude/plugins/outputai/skills/output-eval-audit/SKILL.md

View on GitHub ↗

License Apache-2.0Imported 8/3/2026Version v1

Other skills for the same job

Published skills whose descriptions closely match this one, with their headline benchmark lift.

miosa-osa/eval-auditAudit an LLM evaluation pipeline for correctness, coverage, and reliability. 6 diagnostic areas with structured Check/Finding output. Produces prioritized findings by severity and recommends next skills to run. Catches common eval pitfalls before they corrupt your metrics. Triggers on: "eval audit", "audit evals", "evaluation audit", "check eval pipeline", "eval health"Close match+35 pp pass-rate lift

More skills from the same source and nearby in the registry, with their headline benchmark lift.

Ready to supercharge your AI agents?

See it in action

What's inside

Auditing an Eval Suite

Overview

When to Use

Step 1: Gather Artifacts

Track record

Benchmark

Adoption

SkillSafety

Source

Other skills for the same job

Related skills