Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

Get Started Free →

LLM Evaluation · ancoleman | DecimalAI

ai mlby ancoleman

LLM Evaluation

ancoleman/evaluating-llms

Source ancoleman/ai-design-components ↗License MIT

Evaluate LLM systems using automated metrics, LLM-as-judge, and benchmarks.

SkillScore73Maintained

Automatic

AI Engineer pack

Last verified 8/21/2026

What it changes

Passed 16 of 22 test cases with the skill, and 12 of 22 without it.

Without skill55%

With skill73%

Checks it passed that the model alone missed

Specifies a minimum target score threshold of 0.8 for faithfulness
Sets target score threshold for context recall to greater than 0.8
Ranks bias as the third priority safety metric for customer support

and 3 more in the full report

4 fixed12 passed either way6 still failing0 broke

Pass-rate lift: +18 pp
Tokens: +182%
Agent turns: 0%
Response time: -23%

Model gemini-3.6-flashMeasured 8/21/2026v116 of 22 cases passed with skill

Tokens, turns and time describe resource use, not a dollar price.

Use this skill

Save it into your project's skills folder. Claude Code picks it up in its next session.

curl -fsSL --create-dirs -o .claude/skills/ancoleman-evaluating-llms/SKILL.md https://app.decimal.ai/s/ancoleman-evaluating-llms/SKILL.md

Want it pinned and updatable? pip install "decimalai>=0.15", then decimalai skills add ancoleman-evaluating-llms, and start Claude Code with claude --plugin-dir "$(git rev-parse --show-toplevel)/.agents"

1 · One-time — install the CLI (docs ↗)

pip install "decimalai>=0.15"

2 · Get the skill — writes .agents/skills/ancoleman-evaluating-llms/ and pins it in .decimal/skills.lock

decimalai skills add ancoleman-evaluating-llms

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/ancoleman-evaluating-llms/SKILL.md https://app.decimal.ai/s/ancoleman-evaluating-llms/SKILL.md

Raw file: /s/ancoleman-evaluating-llms/SKILL.md ↗ · machine-readable summary at /s/ancoleman-evaluating-llms.json ↗

Safety checks: 3 of 3 passed

Code scan CleanIntent PassedContent Clean

See it in action

One of 4 test cases the skill fixed. Same model, same prompt, with and without the skill.

1 of 4

Prompt

We are launching a RAG-based knowledge retrieval bot for our enterprise documentation. Which evaluation metric should be prioritized above all others to prevent hallucinated answers, and what minimum score threshold should we enforce before releasing to production?

What was checked, without and with the skill
Checked	Without skill	With skill
Identifies faithfulness as the primary metric to prioritize for hallucination prevention in RAG	Passed	Passed
Specifies a minimum target score threshold of 0.8 for faithfulness	Missed	Passed

case-02 · gemini-3.6-flash · 8/21/2026 · answers are capped and redacted

What's inside

The instructions your model reads when the skill loads.

SKILL.md480 lines8 files

LLM Evaluation

Evaluate Large Language Model (LLM) systems using automated metrics, LLM-as-judge patterns, and standardized benchmarks to ensure production quality and safety.

When to Use This Skill

Apply this skill when:

Testing individual prompts for correctness and formatting
Validating RAG (Retrieval-Augmented Generation) pipeline quality
Measuring hallucinations, bias, or toxicity in LLM outputs
Comparing different models or prompt configurations (A/B testing)
Running benchmark tests (MMLU, HumanEval) to assess model capabilities
Setting up production monitoring for LLM applications
Integrating LLM quality checks into CI/CD pipelines

Common triggers:

"How do I test if my RAG system is working correctly?"
"How can I measure hallucinations in LLM outputs?"
"What metrics should I use to evaluate generation quality?"
"How do I compare GPT-4 vs Claude for my use case?"
"How do I detect bias in LLM responses?"

Evaluation Strategy Selection

Decision Framework: Which Evaluation Approach?

By Task Type:

Task Type	Primary Approach	Metrics	Tools
Classification (sentiment, intent)	Automated metrics	Accuracy, Precision, Recall, F1	scikit-learn
Generation (summaries, creative text)	LLM-as-judge + automated	BLEU, ROUGE, BERTScore, Quality rubric	GPT-4/Claude for judging
Question Answering	Exact match + semantic similarity	EM, F1, Cosine similarity	Custom evaluators
RAG Systems	RAGAS framework	Faithfulness, Answer/Context relevance	RAGAS library
Code Generation	Unit tests + execution	Pass@K, Test pass rate	HumanEval, pytest
Multi-step Agents	Task completion + tool accuracy	Success rate, Efficiency	Custom evaluators

By Volume and Cost:

Track record

Benchmark

16 / 22

test cases passed, vs 12 / 22 without

Adoption

Users: 0
Times used: 0

Safety and source details

SkillSafety

✓ 3 of 3 passed

Code scan14 of 15 checks clean

0 critical · 0 warning · 1 info — scanner v4 — 8/3/2026

View 15 scanner checks

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

No issues (informational notes only).

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

ancoleman / ai-design-components

skills/evaluating-llms/SKILL.md

View on GitHub ↗

License MITImported 8/3/2026Version v1Bundled 7 files

Other skills for the same job

Published skills whose descriptions closely match this one, with their headline benchmark lift.

More skills from the same source and nearby in the registry, with their headline benchmark lift.

Ready to supercharge your AI agents?

See it in action

What's inside

LLM Evaluation

When to Use This Skill

Evaluation Strategy Selection

Decision Framework: Which Evaluation Approach?

Track record

Benchmark

Adoption

SkillSafety

Source

Other skills for the same job

Related skills