Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →yogsoth-ai/benchmark-archaeology
Evaluation Methodology Archaeology Campaign — 5 strategies for systematic analysis of AI/ML benchmarks, metrics, and leaderboards. Reveals construct validity issues, saturation, data contamination, and evaluation protocol inconsistencies.
6 of 22 test cases passed with the skill.
Tokens, turns and time describe resource use, not a dollar price.
Lift withheld: 7 comparable cases; this report requires 8.
One of 3 test cases the skill fixed. Same model, same prompt, with and without the skill.
How many papers should be analyzed during a protocol forensics investigation across 5 target benchmarks?
| Checked | Without skill | With skill |
|---|---|---|
| Specifies a budget of 60 papers | Missed | Passed |
The instructions your model reads when the skill loads.
Systematic excavation and critical analysis of AI/ML evaluation methodology. Treats benchmarks as historical artifacts requiring forensic examination — uncovering hidden assumptions, methodological drift, validity decay, and coverage gaps that accumulate over time.
| Signal | Route To |
|---|---|
| "audit this benchmark", "benchmark quality", "BetterBench" | benchmark-audit |
| "saturation", "ceiling", "score plateau", "when will X be solved" | saturation-analysis |
| "does it actually measure", "construct validity", "what does score mean" | validity-probing |
| "what's not tested", "coverage gaps", "missing capabilities" | coverage-mapping |
| "different papers get different scores", "protocol differences" | protocol-forensics |
| Strategy | Purpose |
|---|---|
| benchmark-audit | Systematic quality assessment using BetterBench 46-criterion framework |
| saturation-analysis | Track score trajectories, detect saturation and failure points |
| validity-probing | Challenge construct validity — does benchmark measure claimed capability? |
| coverage-mapping | Map evaluation coverage, identify untested capability dimensions |
| protocol-forensics | Analyze evaluation protocol differences across papers for same benchmark |
| Tactic | Purpose |
|---|---|
| score-trajectory-analysis | Collect historical scores, fit saturation curves, detect inflection points |
| artifact-detection | Detect annotation artifacts and shortcuts in benchmarks |
| evaluation-protocol-comparison | Compare implementation differences of same benchmark across papers |
| SOP | Purpose |
|---|---|
| benchmark-inventory | Identify and catalog all relevant benchmarks in target domain |
| metric-decomposition | Decompose composite metrics into constituent signals |
test cases passed with the skill
More skills from the same source and nearby in the registry, with their headline benchmark lift.