Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →smithersai/eval-writer
Turn acceptance criteria into a runnable Smithers eval suite (JSONL cases + rubric) and wire it to `smithers eval`.
Passed 22 of 22 test cases with the skill, and 10 of 22 without it.
and 15 more in the full report
12 fixed10 passed either way0 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 12 test cases the skill fixed. Same model, same prompt, with and without the skill.
I want to quickly verify our workflow test suite by running only the first 3 cases instead of all 100 cases. Which command line option caps the number of cases evaluated?
| Checked | Without skill | With skill |
|---|---|---|
| The response specifies the --max-cases flag. | Missed | Passed |
The instructions your model reads when the skill loads.
This skill is the backpressure layer: a suite that tests the claim a workflow is done, since one passing run proves nothing repeatable. It turns acceptance criteria into cases, runs the whole workflow, and exits non-zero on any regression: a gate that can fail, not just "the agent said it works."
It evaluates the model and harness together: it runs the real <Workflow>
(agents, schemas, retries, branches, loops) against fixed inputs and asserts
on the persisted output, never a prompt scored alone. Any regression in
that stack (a worse model, a broken prompt, a dropped field, a mis-wired
branch) reds out a case.
smithers optimize runs a suite twice).Skip it for one-off prompts nothing depends on: backpressure is for behavior you need steady over time.
A suite is a .jsonl file under .smithers/evals/, one case per line: an
input plus an expected assertion. Assertions: status (run finished),
output (exact match), outputContains (partial/deep-subset match, the
usual choice), errorContains (run failed with a matching error substring,
for adversarial cases).
{"id":"happy-path","input":{"prompt":"Draft release notes"},"expected":{"status":"finished"}}
{"id":"lists-breaking-changes","input":{"prompt":"Release notes for v2"},"expected":{"status":"finished","outputContains":{"notes":{"breakingChanges":[{"severity":"high"}]}}}}
Turn each acceptance criterion into ≥1 case: a happy path, the quality-gate
test cases passed, vs 10 / 22 without
More skills from the same source and nearby in the registry, with their headline benchmark lift.