Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →mkurman/lm-evaluation-harness
LLM evaluation framework (EleutherAI lm-evaluation-harness). Unified benchmark evaluation for language models with 200+ tasks, support for HuggingFace transformers, vLLM, SGLang, OpenAI API, GGUF, and custom models. Used by HuggingFace Open LLM Leaderboard. Covers MMLU, HellaSwag, ARC, GSM8K, HumanEval, BBH, TruthfulQA, and more.
Passed 22 of 24 test cases with the skill, and 18 of 24 without it.
and 1 more in the full report
4 fixed18 passed either way2 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 4 test cases the skill fixed. Same model, same prompt, with and without the skill.
I am running a 5-shot evaluation on MMLU with lm-eval CLI and need reproducible shot selection. Which argument sets the seed for sampling few-shot examples?
| Checked | Without skill | With skill |
|---|---|---|
| The response specifies --fewshot_random_seed. | Missed | Passed |
The instructions your model reads when the skill loads.
|
| leaderboard | MMLU, ARC, HellaSwag, TruthfulQA, Winogrande, GSM8K | Open LLM Leaderboard suite |
| mmlu | 57 subjects (STEM, humanities, social sciences) | World knowledge + reasoning |
| gsm8k | Grade-school math word problems | Mathematical reasoning |
| hellaswag | Commonsense NLI | Commonsense reasoning |
| arc_challenge | Science exam questions | Scientific reasoning |
| truthfulqa | Adversarial questions | Truthfulness/hallucination |
| humaneval | Python code generation | Coding ability |
| bigbench | 200+ BIG-Bench tasks | Broad capability assessment |
| ifeval | Instruction-following | Instruction adherence |
Run leaderboard suite:
lm-eval run \
--model hf \
--model_args pretrained=your-model \
--tasks leaderboard \
--device cuda:0 \
--batch_size auto
from lm_eval import simple_evaluate
results = simple_evaluate(
model="hf",
model_args={"pretrained": "meta-llama/Llama-3.2-1B"},
tasks=["mmlu", "hellaswag", "gsm8k"],
device="cuda:0",
batch_size="auto",
limit=100, # Optional: limit samples per task
)
# Access results
for task, metrics in results["results"].items():
print(f"{task}: {metrics}")
# Formatted table
test cases passed, vs 18 / 24 without
Published skills whose descriptions closely match this one, with their headline benchmark lift.
More skills from the same source and nearby in the registry, with their headline benchmark lift.