Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

Get Started Free →

Hugging Face Community Evals · lingxling | DecimalAI

ai mlby lingxling

Hugging Face Community Evals

lingxling/hugging-face-community-evals

Source lingxling/awesome-skills-cn ↗License MIT

Run local evaluations for Hugging Face Hub models with inspect-ai or lighteval.

SkillScore95Maintained

Automatic

AI Engineer pack

Last verified 8/28/2026

What it changes

Passed 19 of 20 test cases with the skill, and 3 of 20 without it.

Without skill14%

With skill95%

Checks it passed that the model alone missed

Command specifies --backend hf
Command specifies --max-samples
Recommends reducing --batch-size

and 20 more in the full report

18 fixed3 passed either way1 still failing0 broke

Pass-rate lift: +82 pp
Tokens: +32%
Agent turns: 0%
Response time: -65%

Model gemini-3.6-flashMeasured 8/28/2026v121 of 22 cases passed with skill · 20 comparable

Tokens, turns and time describe resource use, not a dollar price.

Use this skill

Save it into your project's skills folder. Claude Code picks it up in its next session.

curl -fsSL --create-dirs -o .claude/skills/lingxling-hugging-face-community-evals/SKILL.md https://app.decimal.ai/s/lingxling-hugging-face-community-evals/SKILL.md

Want it pinned and updatable? pip install "decimalai>=0.15", then decimalai skills add lingxling-hugging-face-community-evals, and start Claude Code with claude --plugin-dir "$(git rev-parse --show-toplevel)/.agents"

1 · One-time — install the CLI (docs ↗)

pip install "decimalai>=0.15"

2 · Get the skill — writes .agents/skills/lingxling-hugging-face-community-evals/ and pins it in .decimal/skills.lock

decimalai skills add lingxling-hugging-face-community-evals

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/lingxling-hugging-face-community-evals/SKILL.md https://app.decimal.ai/s/lingxling-hugging-face-community-evals/SKILL.md

Raw file: /s/lingxling-hugging-face-community-evals/SKILL.md ↗ · machine-readable summary at /s/lingxling-hugging-face-community-evals.json ↗

Safety checks: 3 of 3 passed

Code scan CleanIntent PassedContent Clean

See it in action

One of 16 test cases the skill fixed. Same model, same prompt, with and without the skill.

1 of 16

Prompt

When conducting an initial smoke test for lighteval on a local GPU before scaling up full benchmarks, what command argument limits evaluation to 10 samples?

What was checked, without and with the skill
Checked	Without skill	With skill
Command specifies --max-samples 10	Missed	Passed

case-10 · gemini-3.6-flash · 8/28/2026 · answers are capped and redacted

What's inside

The instructions your model reads when the skill loads.

SKILL.md210 lines4 filesincludes scripts

Overview

When to Use

Use this skill for local model evaluation, backend selection, and GPU smoke tests outside the Hugging Face Jobs workflow.

This skill is for running evaluations against models on the Hugging Face Hub on local hardware.

It covers:

inspect-ai with local inference
lighteval with local inference
choosing between vllm, Hugging Face Transformers, and accelerate
smoke tests, task selection, and backend fallback strategy

It does not cover:

Hugging Face Jobs orchestration
model-card or model-index edits
README table extraction
Artificial Analysis imports
.eval_results generation or publishing
PR creation or community-evals automation

If the user wants to run the same eval remotely on Hugging Face Jobs, hand off to the hugging-face-jobs skill and pass it one of the local scripts in this skill.

If the user wants to publish results into the community evals workflow, stop after generating the evaluation run and hand off that publishing step to ~/code/community-evals.

All paths below are relative to the directory containing this SKILL.md.

When To Use Which Script

Use case	Script
Local `inspect-ai` eval on a Hub model via inference providers	`scripts/inspect_eval_uv.py`
Local GPU eval with `inspect-ai` using `vllm` or Transformers	`scripts/inspect_vllm_uv.py`
Local GPU eval with `lighteval` using `vllm` or `accelerate`	`scripts/lighteval_vllm_uv.py`
Extra command patterns	`examples/USAGE_EXAMPLES.md`

Prerequisites

Prefer uv run for local execution.
Set HF_TOKEN for gated/private models.

Track record

Benchmark

19 / 20

test cases passed, vs 3 / 20 without

Adoption

Users: 3
Times used: 337

Safety and source details

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v4 — 8/3/2026

View 15 scanner checks

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

lingxling / awesome-skills-cn

antigravity-awesome-skills/plugins/antigravity-awesome-skills-claude/skills/hugging-face-community-evals/SKILL.md

View on GitHub ↗

License MITImported 8/3/2026Version v1Bundled 3 files · scripts

Other skills for the same job

Published skills whose descriptions closely match this one, with their headline benchmark lift.

More skills from the same source and nearby in the registry, with their headline benchmark lift.