Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →miosa-osa/eval-audit
Audit an LLM evaluation pipeline for correctness, coverage, and reliability. 6 diagnostic areas with structured Check/Finding output. Produces prioritized findings by severity and recommends next skills to run. Catches common eval pitfalls before they corrupt your metrics.
Passed 16 of 21 test cases with the skill, and 9 of 21 without it.
and 14 more in the full report
9 fixed9 passed either way4 still failing1 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 8 test cases the skill fixed. Same model, same prompt, with and without the skill.
We need a fast check on evals/agent_bench/ in our pre-commit hook that skips time-consuming statistical significance calculations and deep sampling checks. Which flag enables this fast mode?
| Checked | Without skill | With skill |
|---|---|---|
| Includes the --quick flag in the audit command execution. | Missed | Passed |
The instructions your model reads when the skill loads.
Audit an LLM evaluation pipeline for correctness and reliability.
Systematically audit an LLM evaluation pipeline across 6 diagnostic areas: data quality, metric validity, judge reliability, coverage gaps, statistical rigor, and pipeline integrity. Each area produces structured Check/Finding pairs with severity ratings. The audit catches common eval pitfalls — label leakage, metric gaming, distribution mismatch, underpowered samples, and judge bias — before they corrupt decision-making. Outputs prioritized recommendations and suggests follow-up skills.
# Full audit of an eval pipeline
/eval-audit --pipeline evals/
# Audit specific diagnostic area
/eval-audit --pipeline evals/ --area judge-reliability
# Audit with custom severity threshold
/eval-audit --pipeline evals/ --min-severity major
# Audit from eval config file
/eval-audit --config evals/config.yaml
# Quick scan (skip deep statistical checks)
/eval-audit --pipeline evals/ --quick
| Flag | Type | Default | Description |
|---|---|---|---|
--pipeline | string | required | Path to evaluation pipeline directory |
--config | string | — | Path to eval config file (alternative to --pipeline) |
--area | enum | all | Diagnostic area: data, metrics, judges, coverage, statistics, integrity, all |
--min-severity | enum | minor | Minimum severity to report: critical, major, minor |
--quick | flag | false | Quick scan (skip computationally expensive checks) |
--output | string | stdout | Write report to file |
--format | enum | markdown | Output format: markdown, json |
test cases passed, vs 9 / 21 without
Published skills whose descriptions closely match this one, with their headline benchmark lift.
More skills from the same source and nearby in the registry, with their headline benchmark lift.