Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

RAG Eval · agentscope-ai | DecimalAI

ai mlby agentscope-ai

RAG Eval

agentscope-ai/rag-eval

Source agentscope-ai/openjudge ↗License Apache-2.0

Use when the user has a RAG (Retrieval-Augmented Generation) system and wants to evaluate its quality — separating retrieval issues from generation issues. Also use when the user mentions RAG evaluation, faithfulness checking, hallucination detection in RAG, retrieval quality, chunking optimization, or "is my RAG pipeline working." Outputs a diagnostic matrix that pinpoints whether problems are in retrieval or generation.

SkillScore83Maintained

Automatic

AI Engineer pack

Last verified 8/15/2026

What it changes

Passed 20 of 24 test cases with the skill, and 15 of 24 without it.

Without skill63%

With skill83%

Checks it passed that the model alone missed

Uses RelevanceGrader to measure query answerability.
Uses HallucinationGrader to measure context grounding.
Passes the --self-test flag to scripts/rag_diagnostic.py.

and 6 more in the full report

7 fixed13 passed either way2 still failing2 broke

Pass-rate lift: +21 pp
Tokens: +68%
Agent turns: 0%
Response time: -36%

Model gemini-3.6-flashMeasured 8/15/2026v120 of 24 cases passed with skill

Tokens, turns and time describe resource use, not a dollar price.

Use this skill

Save it into your project's skills folder. Claude Code picks it up in its next session.

curl -fsSL --create-dirs -o .claude/skills/agentscope-ai-rag-eval/SKILL.md https://app.decimal.ai/s/agentscope-ai-rag-eval/SKILL.md

Want it pinned and updatable? pip install "decimalai>=0.15", then decimalai skills add agentscope-ai-rag-eval, and start Claude Code with claude --plugin-dir "$(git rev-parse --show-toplevel)/.agents"

1 · One-time — install the CLI (docs ↗)

pip install "decimalai>=0.15"

2 · Get the skill — writes .agents/skills/agentscope-ai-rag-eval/ and pins it in .decimal/skills.lock

decimalai skills add agentscope-ai-rag-eval

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/agentscope-ai-rag-eval/SKILL.md https://app.decimal.ai/s/agentscope-ai-rag-eval/SKILL.md

Raw file: /s/agentscope-ai-rag-eval/SKILL.md ↗ · machine-readable summary at /s/agentscope-ai-rag-eval.json ↗

Safety checks: 3 of 3 passed

Code scan CleanIntent PassedContent Clean

See it in action

One of 7 test cases the skill fixed. Same model, same prompt, with and without the skill.

1 of 7

Prompt

Before running our bulk evaluation traces through the diagnostic script, we want to verify that the CLI utility functions correctly on our environment. What command flag performs a built-in verification check?

What was checked, without and with the skill
Checked	Without skill	With skill
Passes the --self-test flag to scripts/rag_diagnostic.py.	Missed	Passed

case-18 · gemini-3.6-flash · 8/15/2026 · answers are capped and redacted

What's inside

The instructions your model reads when the skill loads.

SKILL.md239 lines2 filesincludes scripts

RAG Eval

Evaluate RAG systems by diagnosing retrieval and generation separately. A single "RAG accuracy" number hides whether the problem is finding the right documents or using them correctly. This skill separates them so you know what to fix.

When to Activate

User has a RAG pipeline (retriever + generator) with traces
User wants to know if their RAG system hallucinates
User is optimizing chunking strategy and needs before/after comparison
User wants to build a RAG evaluation dataset

Checklist

You MUST create a task for each item and complete them in order:

Load RAG traces — validate query + context + answer triples
Separate retrieval vs generation — determine which layers to evaluate
Run faithfulness evaluation — is the answer grounded in retrieved docs?
Run retrieval evaluation — are the right documents retrieved?
Build diagnostic matrix — cross-tabulate to find root cause
Output findings — prioritized issues with concrete fixes

Fast path: run the bundled script

Once each trace has a faithfulness judgment (and ideally a retrieval signal), build the retrieval-vs-generation diagnostic matrix with the bundled, tested script (scripts/rag_diagnostic.py, standard library only, no OpenJudge dependency):

python scripts/rag_diagnostic.py --traces traces.jsonl

Trace rows: {"faithful":bool} or {"faithfulness_score":1-5} (>= 4 = faithful), plus an optional retrieval signal {"retrieval_good":bool} or {"recall_at_k":0-1} (>= 0.5 = good). It prints the generation faithful/hallucinating split, the 2×2 matrix when a retrieval signal is present, and the primary issue (retrieval vs generation). --self-test to verify it.

Steps below explain how to separate the layers and produce the faithfulness/retrieval signals

Track record

Benchmark

20 / 24

test cases passed, vs 15 / 24 without

Adoption

Users: 3
Times used: 45

Safety and source details

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v4 — 8/3/2026

View 15 scanner checks

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

agentscope-ai / openjudge

skills/eval_pipeline/05-rag-eval/SKILL.md

View on GitHub ↗

License Apache-2.0Imported 8/3/2026Version v1Bundled 1 file · scripts

More skills from the same source and nearby in the registry, with their headline benchmark lift.

Ready to supercharge your AI agents?

See it in action

What's inside

RAG Eval

When to Activate

Checklist

Fast path: run the bundled script

Track record

Benchmark

Adoption

SkillSafety

Source

Related skills