Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →yogsoth-ai/artifact-detection
Detect annotation artifacts and shortcuts in benchmarks
Passed 21 of 22 test cases with the skill, and 5 of 22 without it.
and 27 more in the full report
17 fixed4 passed either way0 still failing1 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 17 test cases the skill fixed. Same model, same prompt, with and without the skill.
On a factual retrieval benchmark, context-only baselines perform near random chance (gap of 45 points), contrast sets show a minor 4 point drop, and template variations cause only a 3 point change. Render an artifact assessment report in YAML format.
| Checked | Without skill | With skill |
|---|---|---|
| The partial input gap is identified as 45 points. | Passed | Passed |
| The overall_severity field is set to low as artifacts are minor and full context is required. | Missed | Passed |
The instructions your model reads when the skill loads.
Systematically probe benchmarks for annotation artifacts, dataset shortcuts, and spurious correlations that allow models to achieve high scores without the intended capability.
Search literature for evidence that partial-input baselines achieve unexpectedly high performance:
Search queries: "[benchmark] annotation artifacts", "[benchmark] hypothesis only", "[benchmark] spurious correlations", "[benchmark] dataset bias"
If published partial-input results exist, record performance gap between partial and full input. Gap < 10 points above random indicates severe artifacts.
Identify whether contrast sets or adversarial evaluations exist:
Record performance drops on contrast sets. Drops > 20 points indicate reliance on surface patterns.
Search for evidence of format sensitivity:
Record whether minor format changes cause disproportionate score changes.
Aggregate evidence into artifact severity assessment:
test cases passed, vs 5 / 22 without
More skills from the same source and nearby in the registry, with their headline benchmark lift.