Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →majiayu000/agent-evals
Design and implement evaluation frameworks for AI agents.
Passed 18 of 23 test cases with the skill, and 12 of 23 without it.
and 4 more in the full report
6 fixed12 passed either way5 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 6 test cases the skill fixed. Same model, same prompt, with and without the skill.
We need a Python grader function to check whether an AI tutoring agent's output dictionary contains structured feedback. The feedback dictionary should be marked as passed only if it includes strengths, area for improvement, and actionable recommendations. Write a Python function grader_feedback_structure(output: dict) -> dict to perform this check.
| Checked | Without skill | With skill |
|---|---|---|
| Returns a dictionary containing boolean pass checks for individual criteria. | Missed | Passed |
| Sets the overall passed boolean to true only when all individual structural checks pass. | Passed | Passed |
The instructions your model reads when the skill loads.
Core Thesis: "One of the biggest predictors for whether someone is able to build agentic workflows really well is whether or not they're able to drive a really disciplined evaluation process." — Andrew Ng
Evaluations (evals) are exams for your agent's reasoning. Unlike traditional testing (TDD) that checks code correctness with PASS/FAIL outcomes, evals measure reasoning quality with probabilistic scores. The distinction is critical:
| Aspect | TDD (Code Testing) | Evals (Agent Evaluation) |
|---|---|---|
| Tests | Does function return correct output? | Did agent make the right decision? |
| Outcome | PASS or FAIL (deterministic) | Scores (probabilistic) |
| Example | "Does get_weather() return valid JSON?" | "Did agent correctly interpret user intent?" |
| Analogy | Testing if calculator works | Testing if student knows WHEN to use multiplication |
Activate this skill when:
Think of evals like course exams for your agent:
| Eval Type | Analogy | Purpose |
|---|---|---|
| Initial Eval | Final exam | Does agent pass the course? Handles all scenarios? |
| Regression Eval | Pop quiz | Did the update break what was working? |
| Component Eval | Subject test | Test individual skills (routing, tool use, output) |
| End-to-End Eval | Comprehensive exam | Test the full experience |
Evals vary on two dimensions:
test cases passed, vs 12 / 23 without
Published skills whose descriptions closely match this one, with their headline benchmark lift.
More skills from the same source and nearby in the registry, with their headline benchmark lift.