Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →mkurman/lighteval
All-in-one LLM evaluation toolkit (HuggingFace LightEval). 1000+ tasks with multi-backend support: Accelerate, vLLM, SGLang, Nanotron, TGI, LiteLLM, inference providers, and custom models. Sample-by-sample result exploration, custom task/metric creation. Used by HuggingFace's Leaderboard and Evals team. For pure GPT-style model eval, also consider lm-evaluation-harness.
Passed 20 of 22 test cases with the skill, and 9 of 22 without it.
and 16 more in the full report
12 fixed9 passed either way2 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 11 test cases the skill fixed. Same model, same prompt, with and without the skill.
What is the recommended, modern primary CLI subcommand for LightEval when running evaluations with the preferred inspect-ai backend?
| Checked | Without skill | With skill |
|---|---|---|
| Specifies lighteval eval as the subcommand. | Missed | Passed |
The instructions your model reads when the skill loads.
---|---------|----------|
| inspect-ai | lighteval eval | Preferred, modern backend |
| Accelerate | lighteval accelerate | Single/multi-GPU local models |
| vLLM | lighteval vllm | Fast batched inference |
| SGLang | lighteval sglang | Structured generation evals |
| Nanotron | lighteval nanotron | Distributed training evaluation |
| TGI | lighteval endpoint tgi | Locally served models |
| LiteLLM | lighteval endpoint litellm | Any API (OpenAI, Anthropic, etc.) |
| HF Providers | lighteval endpoint inference-providers | HuggingFace's hosted API |
| Inference Endpoints | lighteval endpoint inference-endpoint | HF Dedicated Endpoints |
| Custom | lighteval custom | Anything with a Python API |
Knowledge Tasks:
lighteval accelerate "model-name" mmlu # 57-subject knowledge
lighteval accelerate "model-name" mmlu-pro # Harder MMLU
lighteval accelerate "model-name" gpqa # Graduate-level science
lighteval accelerate "model-name" triviaqa # Trivia QA
lighteval accelerate "model-name" humanitys_last_exam # Very hard questions
Math and Code:
lighteval accelerate "model-name" gsm8k # Grade school math
lighteval accelerate "model-name" math # Competition math
lighteval accelerate "model-name" aime24 # AIME 2024
lighteval accelerate "model-name" lcb # LiveCodeBench
Chat Model Evaluation:
lighteval accelerate "model-name" ifeval # Instruction following
lighteval accelerate "model-name" mt_bench # Multi-turn dialogue
lighteval accelerate "model-name" musr # Multi-step reasoning
lighteval accelerate "model-name" ruler # Long context
Multilingual:
test cases passed, vs 9 / 22 without
Published skills whose descriptions closely match this one, with their headline benchmark lift.
More skills from the same source and nearby in the registry, with their headline benchmark lift.