Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →borghei/agentic-evaluation-framework
This skill should be used when the user asks to "evaluate LLM output quality", "set up LLM-as-judge", "build an eval rubric", "compare model outputs pairwise", or "measure agent quality".
Passed 19 of 23 test cases with the skill, and 12 of 23 without it.
and 5 more in the full report
7 fixed12 passed either way4 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 7 test cases the skill fixed. Same model, same prompt, with and without the skill.
Does the evaluation framework require us to configure API keys for OpenAI or Anthropic inside `scripts/rubric_scorer.py` when running our evaluation aggregation?
| Checked | Without skill | With skill |
|---|---|---|
| Clarifies that scripts/rubric_scorer.py operates entirely offline without making API calls. | Missed | Passed |
The instructions your model reads when the skill loads.
Category: Engineering Domain: AI Engineering
Design and run trustworthy evaluations for LLM and agent outputs: pick the right grading method (programmatic check, LLM-as-judge, or human review), write a scoring rubric that judges can apply consistently, rank competing variants by pairwise comparison, and watch for the biases that quietly corrupt judge scores — position bias, verbosity bias, and self-preference. The goal is an eval that you can trust enough to ship on: calibrated against human labels, cheap enough to run on every change, and tracked alongside cost and latency so you never trade quality away by accident. This skill is model- and vendor-agnostic: it reasons about the evaluation method, not any one provider's API, and its scripts aggregate scores you have already collected — they never call a model.
Before designing or running an evaluation, confirm these inputs. If any is unknown or vague, ASK — do not assume:
criteria and weights)rubric_scorer.py absolute scoring vs pairwise_ranking.py comparison vs human-in-the-loop)Stop rule: ask only the 2-3 that most change the output. If the user says "just draft it," proceed and list your assumptions.
cd engineering/agentic-evaluation-framework
# 1. Score outputs against a weighted rubric + check inter-rater agreement
python scripts/rubric_scorer.py --data rubric_scores.json
# 2. Rank competing variants from pairwise (A-vs-B) judgements
python scripts/pairwise_ranking.py --data pairwise_matches.json
# JSON output for piping into a dashboard or CI gate
python scripts/rubric_scorer.py --data rubric_scores.json --json
| Tool | Purpose | Key Flags |
|---|---|---|
scripts/rubric_scorer.py | Aggregate per-criterion scores into weighted totals, per-criterion means, pass/fail vs thresholds, and an inter-rater agreement metric | --data, --json |
scripts/pairwise_ranking.py | Turn head-to-head win/loss records into a ranking via Elo + Bradley-Terry, plus a win-rate matrix | --data, --k, --base, --json |
test cases passed, vs 12 / 23 without
Published skills whose descriptions closely match this one, with their headline benchmark lift.
More skills from the same source and nearby in the registry, with their headline benchmark lift.