Use when designing or auditing experiments for an ACL paper, covering tuned LLM baselines, multi-dataset and multilingual evaluation, statistical significance and variance, human evaluation with agreement reporting, contamination and prompt-sensitivity controls, ablations, and error-analysis expectations in NLP reviewing.
SkillScore82Maintained
Automatic
◆Research pack
Last verified 8/15/2026
What it changes
Passed 18 of 22 test cases with the skill, and 12 of 22 without it.
Without skill55%
With skill82%
Checks it passed that the model alone missed
The response includes '[Baseline gaps]'.
The response uses the exact bracketed section header '[Evidence verdict]'.
The response includes the exact header '[Evidence verdict] misaligned-with-claim'.
and 6 more in the full report
8 fixed10 passed either way2 still failing2 broke
Pass-rate lift
+27 pp
Tokens
+23%
Agent turns
0%
Response time
-31%
Model gemini-3.6-flashMeasured 8/15/2026v118 of 22 cases passed with skill
Tokens, turns and time describe resource use, not a dollar price.
See it in action
One of 8 test cases the skill fixed. Same model, same prompt, with and without the skill.
1 of 8
Prompt
We have a strict compute budget of 100 GPU hours for our upcoming ACL paper. We cannot afford 5 seeds across all 10 experimental configurations. We plan to pick our best single run retrospectively for the headline table. Critique our compute management strategy using the standard five bracketed review format.
What was checked, without and with the skill
Checked
Without skill
With skill
The response recommends pre-registering which single configuration gets multi-seed treatment as the headline setting.
Missed
Passed
The response suggests paired experimental designs to maximize statistical power under compute constraints.
Missed
Passed
case-21 · gemini-3.6-flash · 8/15/2026 · answers are capped and redacted
What's inside
The instructions your model reads when the skill loads.
SKILL.md120 linessingle file
ACL Experiments
Use this while the experimental story can still change. The ACL evidence bar
is not "beats the baseline once": it is a defensible measurement of a language
capability, with the failure modes examined.
Baseline honesty
Include the strongest cheap baseline: a well-prompted current LLM has become
mandatory context for most tasks — a method beating only pre-LLM systems
invites the "does this matter now?" review.
Tune baselines with the same care as your method (same search budget, same
data); reviewers explicitly probe for asymmetric tuning.
Report the trivial baselines (majority class, copy input, retrieval-only)
when they contextualize how hard the task actually is.
Evaluation design
Breadth must match the claim: a "general" claim needs multiple datasets;
a cross-lingual claim needs typologically distinct languages, not three
Romance neighbors.
Automatic metrics need justification for generation tasks — pair n-gram or
embedding metrics with human or LLM-judge evaluation, and validate any
LLM-judge against human labels before leaning on it.
Fix the evaluation protocol before final runs: dev-set peeking on the test
set via repeated submissions is unreportable and unrepairable.
Statistical floor
Result flavor
Required rigor at ACL
Small deltas between systems
Significance test (bootstrap/permutation) or overlapping-interval honesty
Fine-tuning results
Multiple seeds; mean and deviation in the table, defined in the caption
Prompted-LLM results
Multiple prompt paraphrases and/or samples; sensitivity range reported
Human evaluation
Raters per item, agreement statistic (e.g., Krippendorff's alpha), pay disclosed
Correlation claims (metrics)
Confidence intervals and comparison against existing metric correlations
The Responsible NLP checklist (Section C) asks for descriptive statistics and
error bars — an experiment plan that cannot fill Section C truthfully is
incomplete by construction.