Hugging Face Evaluation

Overview

This skill provides tools to add structured evaluation results to Hugging Face model cards. It supports multiple methods for adding evaluation data:

• Extracting existing evaluation tables from README content
• Importing benchmark scores from Artificial Analysis
• Running custom model evaluations with vLLM or accelerate backends (lighteval/inspect-ai)

When to Use

• You need to add structured evaluation results to a Hugging Face model card.
• You want to import benchmark data or run custom evaluations with vLLM, lighteval, or inspect-ai.
• You are preparing leaderboard-compatible model-index metadata for a model release.

Integration with HF Ecosystem

• Model Cards: Updates model-index metadata for leaderboard integration
• Artificial Analysis: Direct API integration for benchmark imports
• Papers with Code: Compatible with their model-index specification
• Jobs: Run evaluations directly on Hugging Face Jobs with uv integration
• vLLM: Efficient GPU inference for custom model evaluation
• lighteval: HuggingFace's evaluation library with vLLM/accelerate backends
• inspect-ai: UK AI Safety Institute's evaluation framework

Version

1.3.0

Dependencies

Core Dependencies

• huggingface_hub>=0.26.0
• markdown-it-py>=3.0.0
• python-dotenv>=1.2.1
• pyyaml>=6.0.3
• requests>=2.32.5
• re (built-in)

Inference Provider Evaluation

• inspect-ai>=0.3.0
• inspect-evals
• openai

vLLM Custom Model Evaluation (GPU required)

• lighteval[accelerate,vllm]>=0.6.0