Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

Get Started Free →

Huggingface Community Evals · waybarrios | DecimalAI

ai mlby waybarrios

Huggingface Community Evals

waybarrios/huggingface-community-evals

Source waybarrios/opencode-power-pack ↗License MIT

Run evaluations for Hugging Face Hub models using inspect-ai and lighteval on local hardware.

SkillScore95Maintained

Automatic

AI Engineer pack

Last verified 8/19/2026

What it changes

Passed 21 of 22 test cases with the skill, and 13 of 22 without it.

Without skill59%

With skill95%

Checks it passed that the model alone missed

Includes the --trust-remote-code flag
Uses the task format lighteval|hellaswag|0
Recommends running scripts/inspect_eval_uv.py

and 7 more in the full report

9 fixed12 passed either way0 still failing1 broke

Pass-rate lift: +36 pp
Tokens: +36%
Agent turns: 0%
Response time: -60%

Model gemini-3.6-flashMeasured 8/19/2026v121 of 22 cases passed with skill

Tokens, turns and time describe resource use, not a dollar price.

Use this skill

Save it into your project's skills folder. Claude Code picks it up in its next session.

curl -fsSL --create-dirs -o .claude/skills/waybarrios-huggingface-community-evals/SKILL.md https://app.decimal.ai/s/waybarrios-huggingface-community-evals/SKILL.md

Want it pinned and updatable? pip install "decimalai>=0.15", then decimalai skills add waybarrios-huggingface-community-evals, and start Claude Code with claude --plugin-dir "$(git rev-parse --show-toplevel)/.agents"

1 · One-time — install the CLI (docs ↗)

pip install "decimalai>=0.15"

2 · Get the skill — writes .agents/skills/waybarrios-huggingface-community-evals/ and pins it in .decimal/skills.lock

decimalai skills add waybarrios-huggingface-community-evals

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/waybarrios-huggingface-community-evals/SKILL.md https://app.decimal.ai/s/waybarrios-huggingface-community-evals/SKILL.md

Raw file: /s/waybarrios-huggingface-community-evals/SKILL.md ↗ · machine-readable summary at /s/waybarrios-huggingface-community-evals.json ↗

Safety checks: 3 of 3 passed

Code scan CleanIntent PassedContent Clean

See it in action

One of 9 test cases the skill fixed. Same model, same prompt, with and without the skill.

1 of 9

Prompt

What task string format should be passed to lighteval for running 0-shot HellaSwag?

What was checked, without and with the skill
Checked	Without skill	With skill
Uses the task format lighteval\|hellaswag\|0	Missed	Passed

case-11 · gemini-3.6-flash · 8/19/2026 · answers are capped and redacted

What's inside

The instructions your model reads when the skill loads.

SKILL.md202 lines4 filesincludes scripts

Overview

This skill is for running evaluations against models on the Hugging Face Hub on local hardware.

It covers:

inspect-ai with local inference
lighteval with local inference
choosing between vllm, Hugging Face Transformers, and accelerate
smoke tests, task selection, and backend fallback strategy

It does not cover:

Hugging Face Jobs orchestration
model-card or model-index edits
README table extraction
Artificial Analysis imports
.eval_results generation or publishing
PR creation or community-evals automation

If the user wants to run the same eval remotely on Hugging Face Jobs, submit the same script via hf jobs uv run (CLI) or the hf_jobs() MCP tool if configured, for remote GPU execution.

If the user wants to publish results into the community evals workflow, stop after generating the evaluation run and hand that publishing step off to the project's own community-evals workflow.

All paths below are relative to the directory containing this SKILL.md.

When To Use Which Script

Use case	Script
Local `inspect-ai` eval on a Hub model via inference providers	`scripts/inspect_eval_uv.py`
Local GPU eval with `inspect-ai` using `vllm` or Transformers	`scripts/inspect_vllm_uv.py`
Local GPU eval with `lighteval` using `vllm` or `accelerate`	`scripts/lighteval_vllm_uv.py`
Extra command patterns	`examples/USAGE_EXAMPLES.md`

Prerequisites

Prefer uv run for local execution.
Set HF_TOKEN for gated/private models.
For local GPU runs, verify GPU access before starting:

Track record

Benchmark

21 / 22

test cases passed, vs 13 / 22 without

Adoption

Users: 3
Times used: 49

Safety and source details

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v4 — 8/3/2026

View 15 scanner checks

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

waybarrios / opencode-power-pack

skills/huggingface-community-evals/SKILL.md

View on GitHub ↗

License Apache-2.0 (modified; see UPSTREAMS.json)Imported 8/3/2026Version v1Bundled 3 files · scripts

Other skills for the same job

Published skills whose descriptions closely match this one, with their headline benchmark lift.

More skills from the same source and nearby in the registry, with their headline benchmark lift.