Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →mkurman/benchmark-contamination-scan
Scan training data for benchmark contamination — n-gram overlap, embedding similarity, and exact canary detection against 60+ evaluation datasets.
Passed 20 of 22 test cases with the skill, and 11 of 22 without it.
and 9 more in the full report
10 fixed10 passed either way1 still failing1 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 10 test cases the skill fixed. Same model, same prompt, with and without the skill.
In our benchmark contamination summary report, we need a table summarizing hits grouped by benchmark dataset. What exact column names must be used in the benchmark breakdown table?
| Checked | Without skill | With skill |
|---|---|---|
| Table header includes Benchmark, Contaminated, and Action as columns. | Missed | Passed |
The instructions your model reads when the skill loads.
Training on benchmarks is the cardinal sin of ML. This skill scans your training data against a curated exclusion list of 60+ benchmarks using n-gram overlap (fast, high recall) and embedding similarity (slow, high precision).
Mandatory before every training run. Run on pre-training data, instruction tuning data, and RL alignment data.
BENCHMARKS = {
"mmlu": {"name": "MMLU", "source": "huggingface://cais/mmlu"},
"hellaswag": {"name": "HellaSwag", "source": "huggingface://Rowan/hellaswag"},
"gsm8k": {"name": "GSM8K", "source": "huggingface://gsm8k"},
"humaneval": {"name": "HumanEval", "source": "huggingface://openai_humaneval"},
"boolq": {"name": "BoolQ", "source": "huggingface://boolq"},
"arc": {"name": "ARC", "source": "huggingface://ai2_arc"},
"winogrande": {"name": "Winogrande", "source": "huggingface://winogrande"},
"piqa": {"name": "PIQA", "source": "huggingface://piqa"},
"siqa": {"name": "Social IQA", "source": "huggingface://social_i_qa"},
"openbookqa": {"name": "OpenBookQA", "source": "huggingface://openbookqa"},
"squad": {"name": "SQuAD", "source": "huggingface://rajpurkar/squad"},
"natural_questions": {"name": "Natural Questions", "source": "huggingface://natural_questions"},
"triviaqa": {"name": "TriviaQA", "source": "huggingface://trivia_qa"},
"medqa": {"name": "MedQA", "source": "huggingface://bigbio/med_qa"},
"pubmedqa": {"name": "PubMedQA", "source": "huggingface://pubmed_qa"},
# Add all evaluation datasets used in your project
}
from collections import defaultdict
import hashlib
class ContaminationScanner:
test cases passed, vs 11 / 22 without
More skills from the same source and nearby in the registry, with their headline benchmark lift.