Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →smithersai/eval-driven-development
How this repo does eval-driven development (EDD) for Smithers workflows — write the failing suite first, build until green, validate on a holdout, then optimize.
Passed 21 of 22 test cases with the skill, and 6 of 22 without it.
and 20 more in the full report
15 fixed6 passed either way1 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 15 test cases the skill fixed. Same model, same prompt, with and without the skill.
I want to assert that an agent output contains specific key-value pairs without breaking whenever the descriptive commentary changes. Which field in the expected object should be used for flexible output matching?
| Checked | Without skill | With skill |
|---|---|---|
| Uses outputContains within the expected object | Missed | Passed |
The instructions your model reads when the skill loads.
One passing run proves nothing. EDD is the loop that makes a workflow's quality repeatable: encode acceptance criteria as a failing eval suite before building, go green, and never tune against the holdout. A change to a prompt, model, schema, or graph edge is "done" when the suite says so, not when a run looks good.
*-holdout suite.
Holdout cases are never used to tune prompts, pick models, or accept an
optimization — they exist to catch overfitting to the dev split.smithers optimize runs the dev suite twice
(baseline + GEPA-patched) and writes an artifact. Accept it only if the
holdout suite, run with --optimization <artifact.json>, does not
regress.eval command exits 0 when all cases pass, 1 on
any failure, 4 for an invalid case file. Wire the non-zero exit into CI;
a suite nobody runs is documentation.Suites live in .smithers/evals/ as JSONL, one case per line:
<suite>.jsonl — the dev split; tune against this.<suite>-holdout.jsonl — the holdout split; validate only.metadata: split (dev/holdout), category (the
behavior class, e.g. durable-routing), and criterion (the plain-English
acceptance criterion the case encodes)..smithers/evals/<suite>.json by default; sweep runs go to
.smithers/evals/reports/<label>/..smithers/evals/run-sweep.sh <label> [concurrency] runs every seeded
suite (dev + holdout) and prints a per-suite pass summary — the pattern fortest cases passed, vs 6 / 22 without
More skills from the same source and nearby in the registry, with their headline benchmark lift.