Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →smithersai/smithers-eval
Run a workflow over a JSON/JSONL eval suite and write a regression report. Run `smithers eval --help` for usage details.
Passed 19 of 21 test cases with the skill, and 3 of 21 without it.
and 28 more in the full report
17 fixed3 passed either way2 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 16 test cases the skill fixed. Same model, same prompt, with and without the skill.
Execute workflow `entity-extractor` against test file `evals/entities.json`. Process a maximum of 25 test cases and execute 5 test cases concurrently. Generate the CLI command.
| Checked | Without skill | With skill |
|---|---|---|
| The command limits total processed cases using --maxCases 25 | Missed | Passed |
| The command sets batch case concurrency using --concurrency 5 | Passed | Passed |
The instructions your model reads when the skill loads.
Run a workflow over a JSON/JSONL eval suite and write a regression report.
| Name | Type | Required | Description |
|---|---|---|---|
workflow | string | yes | Workflow ID (from smithers workflow list) or path to a .tsx workflow file |
| Flag | Type | Default | Description |
|---|---|---|---|
--cases | string | JSON or JSONL eval case file | |
--suite | string | Stable suite ID used in run IDs and report paths | |
--runLabel | string | Run label appended to eval run IDs; defaults to current UTC timestamp plus a nonce | |
--dryRun | boolean | false | Plan the suite without launching runs |
--concurrency | number | 1 | Number of eval cases to run at once |
--maxCases | number | Run only the first N cases | |
--report | string | Write report JSON to this path | |
--force | boolean | false | Overwrite an existing eval report |
--includeOutput | boolean | true | Include workflow outputs in the report |
--maxConcurrency | number | Per-workflow max task concurrency | |
--root | string | Tool sandbox root directory | |
--log | boolean | true | Enable NDJSON event log file output |
--logDir | string | NDJSON event logs directory | |
--allowNetwork | boolean | false | Allow bash tool network requests |
--maxOutputBytes | number | Max bytes a single tool call can return | |
--toolTimeoutMs | number | Max wall-clock time per tool call in ms | |
--optimization | string | Apply a Smithers optimization artifact while running the eval suite |
test cases passed, vs 3 / 21 without
More skills from the same source and nearby in the registry, with their headline benchmark lift.