Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →brycewang-stanford/colm-experiments
Use when designing or auditing the empirical core of a COLM paper — contamination analysis for evaluation data, fair baselines under matched prompting and compute, pinned model versions and decoding parameters, uncertainty over runs and samples, scaling coverage, and honest reporting of API-model comparisons.
Passed 21 of 22 test cases with the skill, and 15 of 22 without it.
and 3 more in the full report
6 fixed15 passed either way1 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 6 test cases the skill fixed. Same model, same prompt, with and without the skill.
We benchmarked our code generation pipeline against gpt-4o over three months using the default API endpoint without recording dates or specific snapshot strings. Can we report gpt-4o results using just the base model string?
| Checked | Without skill | With skill |
|---|---|---|
| Requires recording specific API snapshot identifiers for evaluated models | Missed | Passed |
The instructions your model reads when the skill loads.
COLM's reviewer pool works on language models daily, which changes what "rigorous" means: the failure modes they hunt are not missing error bars alone but the LM-specific ones — leaked test sets, un-tuned baselines, drifting API models, and decoding settings that quietly decide the result. Build the evaluation so those four attacks fail.
1. Contamination. Any public benchmark plausibly overlaps the pre-training data of any recent model. For each evaluation set, either (a) run and report an overlap analysis against known corpora, (b) use dated or freshly generated items and say so, or (c) state explicitly that contamination could not be assessed and bound what it could explain. Silence is the only unacceptable option — and a contamination paragraph placed by the results, not buried in an appendix, defuses the rebuttal question before it is asked.
2. Baseline fairness. "Our method beats the baseline" means nothing if the baseline got one prompt attempt and your method got fifty. Matched budgets are the rule: same prompt-engineering effort, same number of few-shot exemplars, same decoding search, same token budget. Log the effort spent tuning each system; the appendix table of "prompts tried per method" is disarming in review.
3. Version drift. Closed models change behind stable-looking names. Pin every API model with its full version string and query dates; pin every open model with a checkpoint revision hash; pin the evaluation harness commit, because scoring implementations drift too. A result on an unpinned model is a rumor.
4. Decoding sensitivity. Temperature, top-p, max tokens, stop sequences, and the system prompt are experimental variables. Report them per experiment, and when the headline claim is close, show it holds under at least one alternative decoding configuration.
LM evaluations have several distinct randomness sources; report the one that actually varies in your setup:
| Noise source | When it applies | Report |
test cases passed, vs 15 / 22 without
Published skills whose descriptions closely match this one, with their headline benchmark lift.
More skills from the same source and nearby in the registry, with their headline benchmark lift.