Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →brycewang-stanford/colm-author-response
Use when writing a COLM rebuttal in the OpenReview discussion phase — triaging reviews released in late May, running cache-based follow-up experiments inside the roughly two-and-a-half-week window, answering contamination and baseline-fairness objections with evidence, and writing for the area chair who decides in July.
Passed 17 of 22 test cases with the skill, and 13 of 22 without it.
and 3 more in the full report
5 fixed12 passed either way4 still failing1 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 5 test cases the skill fixed. Same model, same prompt, with and without the skill.
Our team is managing the rebuttal for our COLM paper on API model evaluation. Reviews just opened, so we have approximately 16 days left. Reviewer 1 doubts API model reproducibility and claims benchmark contamination on post-cutoff items. Reviewer 2 posted a two-line generic review that doesn't engage with our method. Reviewer 3 points out that our main results only cover one model family. Organize our rebuttal strategy into the official format. Specifically, structure the output to include: - A timeline header showing days remaining out of the ~18-day period. - A per-reviewer section showing counts across the 4 feedback categories (R1 through R3) plus the crucial point each reviewer needs to hear. - An experiment execution queue listing cache/lightweight tasks first, complete with assigned owners and finish-by deadlines. - A concessions section outlining what points we acknowledge and how our paper claims will be rescoped. - A 3-sentence preamble targeted at the Area Chair summarizing structural changes made during discussion.
| Checked | Without skill | With skill |
|---|---|---|
| Formats output using required bracketed section headers. | Missed | Passed |
| Categorizes reviewer feedback using standard claim bins. | Passed | Passed |
| Places cache-based analyses first in the experiment queue. | Passed | Passed |
The instructions your model reads when the skill loads.
In 2026 the COLM rebuttal ran May 22 to June 8: reviews and the response window opened the same day, giving about eighteen days between first sight of the reviews and the last word you get before the July 8 decision. A rebuttal in that geometry is a project with a plan, not a reply-all reflex.
days 1-2 read only; draft nothing; log every reviewer claim into the four bins
(misreading / evidence gap / scope fight / correct hit — colm-review-process)
days 3-9 run the winnable experiments — cache-based analyses first, single-GPU
or small-API-budget runs second; anything needing full retraining is
out of scope, say so plainly
days 10-13 draft one response per reviewer + a short common preamble for
cross-cutting points; every paragraph anchored to a page, table,
appendix section, or new number
days 14-16 co-author cold read: does each response open with its conclusion?
days 17-18 post with margin; OpenReview deadline-hour uploads are self-harm
The highest-yield experiments are the ones your artifact discipline pre-paid for: a
cached-responses archive lets you re-slice results, add a metric, or test a reviewer
hypothesis in hours without touching a model (colm-artifact-evaluation).
These four LM-specific objections dominate borderline cases; each has a right and a wrong answer shape:
| Objection | Weak answer | Strong answer |
|---|---|---|
| "Benchmark is likely contaminated for these models." | "This benchmark is widely used." | Overlap analysis numbers, or dated-item split showing the effect survives on post-cutoff items |
| "Baselines are under-tuned; better prompting closes the gap." | "We used standard prompts." | The tried-ledger: k prompt variants per system, matched budgets; if the gap narrows, report the narrowed gap |
| "Results are on one model family; is this about LMs or about model X?" | "We expect it generalizes." | One added family inside the window, claim rescoped in the revision either way |
| "API model results won't reproduce." | "We used the public API." | Version strings + query dates already in App. A, plus commitment to release the response cache |
Notice the pattern: every strong answer is either evidence you can generate this week or a scope concession that makes the paper truer. Both move ACs; indignation
test cases passed, vs 13 / 22 without
Published skills whose descriptions closely match this one, with their headline benchmark lift.
More skills from the same source and nearby in the registry, with their headline benchmark lift.