---
name: smithersai-smithers-eval
description: "Run a workflow over a JSON/JSONL eval suite and write a regression report. Run `smithers eval --help` for usage details."
license: "MIT"
metadata:
  decimalai-skill-id: "1104ed12-60a3-495c-911d-9dede4b8a0b1"
  decimalai-version-id: "8a4af39a-e85f-4adb-ada1-e9aab17c34d4"
  decimalai-version: "1"
  decimalai-body-sha256: "aef9cda195c4639c867bfa223d4c80d7214ccc4335cba4fdd49d3186199c48f7"
  decimalai-name: "smithersai/smithers-eval"
  decimalai-format: "1"
  decimalai-invocation: "model"
  decimalai-category: "ai-ml"
  decimalai-source: "https://app.decimal.ai/s/smithersai-smithers-eval@1/SKILL.md"
---
# smithers eval

Run a workflow over a JSON/JSONL eval suite and write a regression report.

## Arguments

| Name | Type | Required | Description |
|------|------|----------|-------------|
| `workflow` | `string` | yes | Workflow ID (from `smithers workflow list`) or path to a .tsx workflow file |

## Options

| Flag | Type | Default | Description |
|------|------|---------|-------------|
| `--cases` | `string` |  | JSON or JSONL eval case file |
| `--suite` | `string` |  | Stable suite ID used in run IDs and report paths |
| `--runLabel` | `string` |  | Run label appended to eval run IDs; defaults to current UTC timestamp plus a nonce |
| `--dryRun` | `boolean` | `false` | Plan the suite without launching runs |
| `--concurrency` | `number` | `1` | Number of eval cases to run at once |
| `--maxCases` | `number` |  | Run only the first N cases |
| `--report` | `string` |  | Write report JSON to this path |
| `--force` | `boolean` | `false` | Overwrite an existing eval report |
| `--includeOutput` | `boolean` | `true` | Include workflow outputs in the report |
| `--maxConcurrency` | `number` |  | Per-workflow max task concurrency |
| `--root` | `string` |  | Tool sandbox root directory |
| `--log` | `boolean` | `true` | Enable NDJSON event log file output |
| `--logDir` | `string` |  | NDJSON event logs directory |
| `--allowNetwork` | `boolean` | `false` | Allow bash tool network requests |
| `--maxOutputBytes` | `number` |  | Max bytes a single tool call can return |
| `--toolTimeoutMs` | `number` |  | Max wall-clock time per tool call in ms |
| `--optimization` | `string` |  | Apply a Smithers optimization artifact while running the eval suite |