Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

Get Started Free →

Huggingface Community Evals · sediman-agent | DecimalAI

ai mlby sediman-agent

Huggingface Community Evals

sediman-agent/huggingface-community-evals

Source sediman-agent/openskynet ↗License NOASSERTION

Run evaluations for Hugging Face Hub models using inspect-ai and lighteval on local hardware.

SkillScore100Maintained

Automatic

AI Engineer pack

Last verified 8/20/2026

What it changes

Passed 21 of 21 test cases with the skill, and 7 of 21 without it.

Without skill32%

With skill100%

Checks it passed that the model alone missed

The command specifies --task gsm8k.
The command includes --trust-remote-code.
The command specifies --backend accelerate.

and 24 more in the full report

15 fixed7 passed either way0 still failing0 broke

Pass-rate lift: +68 pp
Tokens: +22%
Agent turns: 0%
Response time: -60%

Model gemini-3.6-flashMeasured 8/20/2026v122 of 22 cases passed with skill · 21 comparable

Tokens, turns and time describe resource use, not a dollar price.

Use this skill

Save it into your project's skills folder. Claude Code picks it up in its next session.

curl -fsSL --create-dirs -o .claude/skills/sediman-agent-huggingface-community-evals/SKILL.md https://app.decimal.ai/s/sediman-agent-huggingface-community-evals/SKILL.md

Want it pinned and updatable? pip install "decimalai>=0.15", then decimalai skills add sediman-agent-huggingface-community-evals, and start Claude Code with claude --plugin-dir "$(git rev-parse --show-toplevel)/.agents"

1 · One-time — install the CLI (docs ↗)

pip install "decimalai>=0.15"

2 · Get the skill — writes .agents/skills/sediman-agent-huggingface-community-evals/ and pins it in .decimal/skills.lock

decimalai skills add sediman-agent-huggingface-community-evals

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/sediman-agent-huggingface-community-evals/SKILL.md https://app.decimal.ai/s/sediman-agent-huggingface-community-evals/SKILL.md

Raw file: /s/sediman-agent-huggingface-community-evals/SKILL.md ↗ · machine-readable summary at /s/sediman-agent-huggingface-community-evals.json ↗

Safety checks: 3 of 3 passed

Code scan CleanIntent PassedContent Clean

See it in action

One of 14 test cases the skill fixed. Same model, same prompt, with and without the skill.

1 of 14

Prompt

I want to run a local 20-sample GSM8K evaluation on `meta-llama/Llama-3.2-1B` using local GPU acceleration with inspect-ai. What exact command should I run?

What was checked, without and with the skill
Checked	Without skill	With skill
The command executes scripts/inspect_vllm_uv.py.	Missed	Passed
The command specifies --task gsm8k.	Missed	Passed
The command sets --limit 20.	Passed	Passed

case-11 · gemini-3.6-flash · 8/20/2026 · answers are capped and redacted

What's inside

The instructions your model reads when the skill loads.

SKILL.md202 lines4 filesincludes scripts

Overview

This skill is for running evaluations against models on the Hugging Face Hub on local hardware.

It covers:

inspect-ai with local inference
lighteval with local inference
choosing between vllm, Hugging Face Transformers, and accelerate
smoke tests, task selection, and backend fallback strategy

It does not cover:

Hugging Face Jobs orchestration
model-card or model-index edits
README table extraction
Artificial Analysis imports
.eval_results generation or publishing
PR creation or community-evals automation

If the user wants to run the same eval remotely on Hugging Face Jobs, hand off to the hugging-face-jobs skill and pass it one of the local scripts in this skill.

If the user wants to publish results into the community evals workflow, stop after generating the evaluation run and hand off that publishing step to ~/code/community-evals.

All paths below are relative to the directory containing this SKILL.md.

When To Use Which Script

Use case	Script
Local `inspect-ai` eval on a Hub model via inference providers	`scripts/inspect_eval_uv.py`
Local GPU eval with `inspect-ai` using `vllm` or Transformers	`scripts/inspect_vllm_uv.py`
Local GPU eval with `lighteval` using `vllm` or `accelerate`	`scripts/lighteval_vllm_uv.py`
Extra command patterns	`examples/USAGE_EXAMPLES.md`

Prerequisites

Prefer uv run for local execution.
Set HF_TOKEN for gated/private models.
For local GPU runs, verify GPU access before starting:

Track record

Benchmark

21 / 21

test cases passed, vs 7 / 21 without

Adoption

Users: 8
Times used: 175

Safety and source details

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v4 — 8/3/2026

View 15 scanner checks

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

sediman-agent / openskynet

skills/huggingface_skills/huggingface-community-evals/SKILL.md

View on GitHub ↗

License NOASSERTIONImported 8/3/2026Version v1Bundled 3 files · scripts

Other skills for the same job

Published skills whose descriptions closely match this one, with their headline benchmark lift.

More skills from the same source and nearby in the registry, with their headline benchmark lift.