Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →miosa-osa/eval-rag
Evaluate retrieval and generation quality in RAG pipelines. Separate scoring for retrieval (recall, precision, MRR) and generation (faithfulness, relevance, completeness). End-to-end pipeline assessment with bottleneck identification.
Passed 17 of 18 test cases with the skill, and 11 of 18 without it.
and 11 more in the full report
8 fixed13 passed either way1 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 4 test cases the skill fixed. Same model, same prompt, with and without the skill.
What is the default top-K document retrieval depth evaluated by /eval-rag when the --k parameter is omitted from the command?
| Checked | Without skill | With skill |
|---|---|---|
| Identifies 5 as the default retrieval depth K | Missed | Passed |
The instructions your model reads when the skill loads.
Evaluate retrieval + generation quality in RAG pipelines.
Assess a Retrieval-Augmented Generation pipeline by evaluating retrieval and generation independently, then measuring end-to-end quality. Retrieval scoring checks if the right documents are fetched (recall, precision, MRR, NDCG). Generation scoring checks if the answer is faithful to retrieved context (no hallucination), relevant to the question, and complete. Identifies whether failures originate in retrieval, generation, or both, so you know where to invest improvement effort.
# Full RAG evaluation
/eval-rag --pipeline rag/ --queries eval/queries.jsonl --golden eval/golden.jsonl
# Evaluate retrieval only
/eval-rag --pipeline rag/ --queries eval/queries.jsonl --golden eval/golden.jsonl --stage retrieval
# Evaluate generation only (with pre-fetched contexts)
/eval-rag --contexts retrieved.jsonl --queries eval/queries.jsonl --golden eval/golden.jsonl --stage generation
# Custom retrieval depth
/eval-rag --pipeline rag/ --queries eval/queries.jsonl --golden eval/golden.jsonl --k 10
# Compare two retrieval configurations
/eval-rag --pipeline rag-v1/ --pipeline-b rag-v2/ --queries eval/queries.jsonl --golden eval/golden.jsonl
| Flag | Type | Default | Description |
|---|---|---|---|
--pipeline | string | required | Path to RAG pipeline configuration or directory |
--pipeline-b | string | — | Second pipeline for A/B comparison |
--queries | string | required | Path to evaluation queries (JSONL) |
--golden | string | required | Path to golden answers with relevant doc IDs (JSONL) |
--contexts | string | — | Pre-retrieved contexts (skips retrieval stage) |
--stage | enum | both | Evaluate: retrieval, generation, both |
--k | int | 5 | Retrieval depth (top-K documents) |
--output | string | stdout | Write report to file |
--format | enum | markdown | Output format: markdown, json |
test cases passed, vs 11 / 18 without
More skills from the same source and nearby in the registry, with their headline benchmark lift.