Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →brycewang-stanford/colm-supplementary
Use when deciding what goes into a COLM paper's appendices and supplementary material versus the strict 9-page main text — verbatim prompts, full evaluation configurations, per-task result tables, contamination analyses, human-evaluation protocols, and anonymized code/data packages that survive double-blind review.
Passed 21 of 24 test cases with the skill, and 17 of 24 without it.
and 1 more in the full report
4 fixed17 passed either way3 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 4 test cases the skill fixed. Same model, same prompt, with and without the skill.
In our language model evaluation paper for COLM, we recorded decoding hyperparameters such as temperature and top-p for every experiment group across 30 table cells. A co-author suggests omitting these details from the paper and only uploading the code repository. How should these decoding configurations be structured and placed in the paper artifacts?
| Checked | Without skill | With skill |
|---|---|---|
| Recommends placing decoding configurations in an appendix table | Passed | Passed |
| Specifies providing one row per reported experimental table cell group | Missed | Passed |
The instructions your model reads when the skill loads.
The 2026 format gives you a strict 9-page main text, unlimited citation pages, and appendix space whose guaranteed audience is nobody — reviewers may read it, the rebuttal may point into it, but no rule obliges anyone to open it. That asymmetry is the whole design principle: the main text carries every load-bearing element, and the supplement carries the audit trail that makes the main text checkable.
Ask of each artifact: is this needed to believe the claim, or to re-run it? Believe → main text. Re-run → appendix/supplement. Both → summarize in the main text, deposit fully in the appendix, and cross-reference in both directions.
| Artifact | Placement | Note |
|---|---|---|
| Representative prompt (or skeleton) | Main text | Readers cannot judge an LM evaluation without seeing how the model was asked |
| Full prompt set, verbatim, incl. system prompts | Appendix | Paraphrased prompts are not reproducible; include exact strings |
| Decoding configs per experiment | Appendix table | One row per reported table cell group (colm-experiments) |
| Contamination analysis method + results | Summary in main text; full in appendix | The conclusion is load-bearing; the n-gram machinery is audit trail |
| Per-task / per-model full result grids | Appendix | Main text shows the aggregate + the interesting slice |
| Human-eval protocol: instructions, pay, demographics, agreement | Appendix | Screenshots of annotator UI earn trust |
| Failure cases and transcripts | Appendix, curated | A dozen representative transcripts beat 400 raw ones |
| Code + data package | Supplementary upload / anonymized repo | Must be anonymous at submission (see below) |
| Raw API responses archive | Release artifact (acceptance) | Often too large for review upload; describe it and commit to release |
Three appendix sections repeatedly decide borderline LM papers:
test cases passed, vs 17 / 24 without
Published skills whose descriptions closely match this one, with their headline benchmark lift.
More skills from the same source and nearby in the registry, with their headline benchmark lift.