Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →nousresearch/evaluating-llms-harness
lm-eval-harness: benchmark LLMs (MMLU, GSM8K, etc.).
Passed 22 of 22 test cases with the skill, and 19 of 22 without it.
3 fixed19 passed either way0 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 3 test cases the skill fixed. Same model, same prompt, with and without the skill.
I installed lm-eval via pip and want to view the full list of supported task names directly from the command line tool. What command should I execute?
| Checked | Without skill | With skill |
|---|---|---|
| The output specifies lm-eval ls tasks. | Missed | Passed |
The instructions your model reads when the skill loads.
Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.
lm-evaluation-harness evaluates LLMs across 60+ academic benchmarks using standardized prompts and metrics.
Installation:
pip install lm-eval
Evaluate any HuggingFace model:
lm_eval --model hf \
--model_args pretrained=meta-llama/Llama-2-7b-hf \
--tasks mmlu,gsm8k,hellaswag \
--device cuda:0 \
--batch_size 8
View available tasks:
lm-eval ls tasks
Evaluate model on core benchmarks (MMLU, GSM8K, HumanEval).
Copy this checklist:
Benchmark Evaluation:
- [ ] Step 1: Choose benchmark suite
test cases passed, vs 19 / 22 without
Published skills whose descriptions closely match this one, with their headline benchmark lift.
More skills from the same source and nearby in the registry, with their headline benchmark lift.