mkurman/lm-evaluation-harness

| | leaderboard | MMLU, ARC, HellaSwag, TruthfulQA, Winogrande, GSM8K | Open LLM Leaderboard suite | | mmlu | 57 subjects (STEM, humanities, social sciences) | World knowledge + reasoning | | gsm8k | Grade-school math word problems | Mathematical reasoning | | hellaswag | Commonsense NLI | Commonsense reasoning | | arc_challenge | Science exam questions | Scientific reasoning | | truthfulqa | Adversarial questions | Truthfulness/hallucination | | humaneval | Python code generation | Coding ability | | bigbench | 200+ BIG-Bench tasks | Broad capability assessment | | ifeval | Instruction-following | Instruction adherence |

Run leaderboard suite:

bash
lm-eval run \
    --model hf \
    --model_args pretrained=your-model \
    --tasks leaderboard \
    --device cuda:0 \
    --batch_size auto

4. Python API

python
from lm_eval import simple_evaluate

results = simple_evaluate(
    model="hf",
    model_args={"pretrained": "meta-llama/Llama-3.2-1B"},
    tasks=["mmlu", "hellaswag", "gsm8k"],
    device="cuda:0",
    batch_size="auto",
    limit=100,  # Optional: limit samples per task
)

# Access results
for task, metrics in results["results"].items():
    print(f"{task}: {metrics}")

# Formatted table
print(results["configs"])
print(results["samples"])  # Per-sample outputs

5. Evaluating LoRA / PEFT Adapters

bash
lm-eval run \
    --model hf \
    --model_args pretrained=meta-llama/Llama-3.2-1B,peft=/path/to/lora_adapter \
    --tasks mmlu \
    --device cuda:0

In Python:

python
from lm_eval import simple_evaluate

results = simple_evaluate(
    model="hf",
    model_args={
        "pretrained": "meta-llama/Llama-3.2-1B",
        "peft": "/path/to/lora_adapter",
    },
    tasks=["mmlu"],
)

6. Multi-GPU Evaluation

Data-parallel (model fits on single GPU):

bash
accelerate launch -m lm_eval \
    --model hf \
    --model_args pretrained=model-name \
    --tasks lambada_openai,arc_easy \
    --batch_size 16

Model-parallel (model too large for one GPU):

bash
lm-eval run \
    --model hf \
    --model_args pretrained=model-name,parallelize=True \
    --tasks mmlu \
    --batch_size 8

Both (data + model parallel):

bash
accelerate launch --multi_gpu --num_processes 4 \
    -m lm_eval \
    --model hf \
    --model_args pretrained=model-name,parallelize=True \
    --tasks mmlu \
    --batch_size 8

7. API Model Evaluation

OpenAI-compatible API:

bash
export OPENAI_API_KEY=your-key

lm-eval run \
    --model openai-completions \
    --model_args model=gpt-4o,base_url=https://api.openai.com/v1/completions \
    --tasks mmlu \
    --batch_size 32

Local server (vLLM served):

bash
lm-eval run \
    --model local-completions \
    --model_args model=local-model,base_url=http://localhost:8000/v1/completions \
    --tasks mmlu

8. Custom Tasks (YAML Config)

Create a custom task at lm_eval/tasks/my_task/my_task.yaml:

yaml
task: my_custom_task
dataset_path: my-dataset
dataset_name: default
output_type: multiple_choice
training_split: train
validation_split: validation
doc_to_text: "Question: {{question}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:"
doc_to_target: "{{answer}}"
doc_to_choice: "{{choices}}"
metric_list:
  - metric: acc

Run it:

bash
lm-eval run --model hf --model_args pretrained=model-name --tasks my_custom_task

9. Few-Shot Configuration

bash
lm-eval run \
    --model hf \
    --model_args pretrained=model-name \
    --tasks mmlu \
    --num_fewshot 5 \
    --fewshot_random_seed 42

10. Logging and Output Formats

bash
# JSON output
lm-eval run --model hf --model_args pretrained=model-name \
    --tasks mmlu --output_path results/

# W&B logging
lm-eval run --model hf --model_args pretrained=model-name \
    --tasks mmlu --wandb_args project=eval-runs

Key Evaluation Tips

1. Use --batch_size auto — automatic batch size detection maximizes throughput
2. Always set a --seed for reproducibility across evaluation runs
3. Prefer vLLM backend for models >7B — 5-10x faster than HF
4. Use --limit 100 during development to test task setup quickly
5. mmlu uses 5-shot by default, while most tasks are 0-shot
6. Install model backends separately — base package is lightweight by design
7. GGUF eval requires explicit tokenizer path — skip this and loading may hang

References

• Full CLI Reference
• Configuration Guide
• Python API Documentation
• Task Guide
• Open LLM Leaderboard