Use when designing or auditing AISTATS experiments, simulations, baselines, statistical tests, uncertainty estimates, ablations, random seeds, hyperparameters, compute, dataset handling, and claim-to-evidence fit, with emphasis on experiments that validate theorems rather than chase leaderboards.
SkillScore91Maintained
Automatic
◆Research pack
Last verified 8/15/2026
What it changes
Passed 21 of 23 test cases with the skill, and 17 of 23 without it.
Without skill74%
With skill91%
Checks it passed that the model alone missed
The response explains that quadratic scaling arises from computing or inverting a d x d covariance matrix.
The response specifies a log-log plot of error versus sample size n with fitted slope to verify the theoretical convergence rate.
The output formats the review using the required square-bracketed block structure rather than markdown tables or numeric scorecards.
and 1 more in the full report
4 fixed17 passed either way2 still failing0 broke
Pass-rate lift
+17 pp
Tokens
-4%
Agent turns
0%
Response time
-33%
Model gemini-3.6-flashMeasured 8/15/2026v121 of 23 cases passed with skill
Tokens, turns and time describe resource use, not a dollar price.
See it in action
One of 4 test cases the skill fixed. Same model, same prompt, with and without the skill.
1 of 4
Prompt
I need a pre-submission review of our experimental evaluation section. Please analyze our current results and output an audit covering claim-to-evidence mappings, statistical reporting omissions, reproducibility gaps regarding hardware and hyperparameters, an overall experimental readiness classification, and the top priority simulation or run to perform before locking the paper.
What was checked, without and with the skill
Checked
Without skill
With skill
The output structures the evaluation using the designated square-bracketed key fields rather than standard conference peer-review templates.
Missed
Passed
case-03 · gemini-3.6-flash · 8/15/2026 · answers are capped and redacted
What's inside
The instructions your model reads when the skill loads.
SKILL.md59 linessingle file
AISTATS Experiments
Use this before submission when the empirical or simulation story is not yet locked.
Experiment audit
Map each empirical claim to a table, figure, simulation, ablation, or robustness check.
Include baselines that represent both ML practice and relevant statistical methods.
Separate synthetic simulations that validate assumptions from real-data experiments that
show practical relevance.
Report uncertainty for stochastic results: repeated runs, standard errors, confidence
intervals, paired tests, or bootstrap intervals when appropriate.
Report dataset splits, preprocessing, metrics, hyperparameter search ranges, final chosen
settings, selection criteria, random seeds, hardware, software versions, and runtime.
Add ablations for the mechanism, not just cosmetic variants.
Audit for leakage, selection bias, multiple-comparison issues, and mismatch between
theoretical assumptions and empirical setup.
What experiments are for at this venue
AISTATS experiments exist to validate theory, not to win leaderboards. One focused
simulation confirming a predicted rate outweighs five extra benchmark datasets.
The strongest design triad: a synthetic study where assumptions hold exactly, a study where
they are deliberately violated, and a real-data study showing practical behavior.
Reviewers, frequently statisticians, check whether the empirical regime — sample size,
dimension, noise level — matches the asymptotic regime of the theorems. A bound proven as
n grows but tested only at n = 500 invites the question of relevance.
Theory-validation design table
Theoretical claim
Matching experiment
Reject pattern avoided
Convergence rate in n
Log-log error versus n with fitted slope
"Rates asserted but never plotted"
Confidence-interval coverage
Empirical coverage across many replications
"Nominal 95 percent never verified"
Regret bound
Cumulative regret versus horizon, with the bound curve overlaid
"Bound and trajectory never compared"
Robustness to misspecification
Violation-severity sweep
"Guarantees hold under assumptions the experiments quietly break"
Vignette: a kernel conditional independence test
Suppose the paper proves finite-sample type-I error control under a boundedness assumption.
Use when designing or auditing AAMAS experiments - self-play and population-based training, opponent selection, equilibrium and regret metrics, game-theoretic simulations, ablations, seeds, hyperparameters, compute, and claim-to-evidence fit - with emphasis on experiments that probe the interaction rather than chase a single-agent leaderboard.