Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →brycewang-stanford/ai-model-benchmarking
Benchmark AI models across 60+ academic evaluation suites and metrics
Passed 20 of 21 test cases with the skill, and 17 of 21 without it.
and 2 more in the full report
3 fixed18 passed either way1 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 2 test cases the skill fixed. Same model, same prompt, with and without the skill.
I am finalizing our research manuscript and need to construct a clean Markdown results table for our standard benchmarking section. Could you draft a Markdown table comparing our 7B model against a baseline across primary evaluation benchmarks (general knowledge, grade school math, coding, commonsense reasoning)? In addition to the table showing score differences, include the proper caption/notes block detailing standard few-shot choices and seed averaging rules used for publication.
| Checked | Without skill | With skill |
|---|---|---|
| formats score improvements as positive delta values inside parentheses within model result cells | Missed | Passed |
| specifies in notes that results report the mean across at least 3 random seeds | Passed | Passed |
The instructions your model reads when the skill loads.
Rigorous evaluation is the backbone of machine learning research. A model is only as credible as its evaluation protocol: which benchmarks were used, how metrics were computed, whether results are reproducible, and how they compare to baselines. The proliferation of LLMs has made this both more important and more complex, with over 60 established benchmarks and a rapidly evolving landscape.
This guide covers the practical side of model benchmarking: how to use the EleutherAI Language Model Evaluation Harness (lm-evaluation-harness), how to select benchmarks for different research claims, how to avoid common evaluation pitfalls, and how to present results for publication. The focus is on academic rigor rather than leaderboard chasing.
Whether you are evaluating a fine-tuned model for a paper, comparing architectures for an ablation study, or reviewing a submitted manuscript's evaluation section, these patterns will help ensure the evaluation is sound.
The EleutherAI lm-evaluation-harness is the de facto standard for LLM evaluation in academic research, supporting 60+ tasks and used by most major LLM papers.
# Install
pip install lm-eval
# Run a single benchmark
lm_eval --model hf \
--model_args pretrained=meta-llama/Llama-2-7b-hf \
--tasks mmlu \
--batch_size auto \
--output_path results/llama2-7b/
# Run multiple benchmarks
lm_eval --model hf \
--model_args pretrained=meta-llama/Llama-2-7b-hf \
--tasks mmlu,hellaswag,arc_challenge,winogrande,truthfulqa_mc2 \
--batch_size auto \
--num_fewshot 5 \
--output_path results/llama2-7b/
import lm_eval
test cases passed, vs 17 / 21 without
Published skills whose descriptions closely match this one, with their headline benchmark lift.
More skills from the same source and nearby in the registry, with their headline benchmark lift.