---
name: leoyeai/eval-skills
source: https://app.decimal.ai/s/leoyeai-eval-skills@1/SKILL.md
source_sha256: f00b4ed7285f
---

# eval-skills

AI Agent Skill unit testing framework — a framework-agnostic toolkit for discovering, scaffolding, selecting, evaluating, and reporting on AI skills.

This skill fills the **L1 (Skill Unit Test)** gap that LangSmith / DeepEval leave open: while those platforms focus on agent-level and trajectory-level evaluation (L2-L3), eval-skills targets the individual skill level, ensuring each building block meets quality standards before it ever enters an agent pipeline.

## When to Use This Skill

- **Before deploying a new skill to production** — run `eval` to verify it meets your quality gate.
- **When choosing between multiple candidate skills** — run `select` to rank them on the same benchmark.
- **When a skill is upgraded** — run `report diff` to detect regressions.
- **In CI/CD** — use `--exit-on-fail` to block merges that degrade skill quality.
- **When bootstrapping a new skill** — run `create` to generate a ready-to-fill skeleton.

## Capabilities

### 1. Find Skills

Search for existing skills by keyword, tag, or adapter type.

```bash
eval-skills find \
  --query "web search" \
  --tag retrieval api \
  --adapter http \
  --min-completion 0.8 \
  --skills-dir ./skills \
  --limit 10
```

| Option | Description | Default |
|--------|-------------|---------|
| `-q, --query <string>` | Keyword search (matches name, description, tags) | — |
| `-t, --tag <tags...>` | Filter by tags (intersection: skill must have ALL specified tags) | — |
| `-a, --adapter <type>` | Filter by adapter type (`http`, `subprocess`, `mcp`) | — |
| `--min-completion <rate>` | Minimum historical completion rate (0.0 ~ 1.0) | — |
| `--skills-dir <dir>` | Directory to scan for `skill.json` files | `./skills` |
| `--limit <n>` | Maximum number of results | `20` |

Results are ranked by search relevance (when `--query` is provided) or by historical completion rate (descending).

### 2. Create Skills

Generate a skill skeleton from a template to bootstrap development.

```bash
eval-skills create \
  --name my_api_skill \
  --from-template http_request \
  --output-dir ./skills \
  --description "Fetches weather data from OpenWeather API"
```

| Option | Description | Default |
|--------|-------------|---------|
| `--name <name>` | **Required.** Skill name | — |
| `--from-template <tpl>` | Template type: `http_request`, `python_script`, `mcp_tool` | `http_request` |
| `--output-dir <dir>` | Output directory | `./skills` |
| `--description <text>` | Human-readable description embedded in `skill.json` | — |

Generated file structure:

```
skills/my_api_skill/
  skill.json            # Skill metadata (id, schemas, adapter config)
  adapter.config.json   # Adapter-specific configuration
  tests/
    basic.eval.json     # A starter benchmark with one sample task
  skill.py              # (python_script template only) JSON-RPC entrypoint
```

### 3. Evaluate Skills

Run benchmark evaluations against one or more skills. This is the core command.

```bash
eval-skills eval \
  --skills ./skills/calculator/skill.json ./skills/search/ \
  --benchmark coding-easy \
  --concurrency 4 \
  --timeout 30000 \
  --retries 2 \
  --runs 3 \
  --evaluator exact \
  --format json markdown html \
  --output-dir ./reports \
  --exit-on-fail --min-completion 0.8 \
  --store ./eval-skills.db
```

| Option | Description | Default |
|--------|-------------|---------|
| `--skills <paths...>` | **Required.** Skill file(s) or directory(ies) | — |
| `--benchmark <id\|path>` | Built-in benchmark ID or path to `benchmark.json` | `coding-easy` |
| `--tasks <file>` | Custom tasks JSON file (replaces benchmark) | — |
| `--concurrency <n>` | Number of parallel task executions | `4` |
| `--timeout <ms>` | Per-task timeout in milliseconds | `30000` |
| `--retries <n>` | Retry count on task failure (with incremental backoff) | `0` |
| `--runs <n>` | Repeat evaluation N times for consistency scoring | `1` |
| `--evaluator <type>` | Default scorer type (see Scorer Types below) | `exact` |
| `--format <formats...>` | Output formats: `json`, `markdown`, `html` | `json markdown` |
| `--output-dir <dir>` | Report output directory | `./reports` |
| `--exit-on-fail` | Exit with code 1 if any skill falls below threshold | disabled |
| `--min-completion <rate>` | Threshold for `--exit-on-fail` | `0.7` |
| `--dry-run` | Validate configuration only; do not execute tasks | disabled |
| `--benchmarks-dir <dir>` | Directory containing built-in benchmarks | `./benchmarks` |
| `--store <path>` | SQLite database path for persistent result storage | `./eval-skills.db` |
| `-c, --config <path>` | Path to `eval-skills.config.yaml` | auto-detected |

**Evaluation flow:**

1. Load skills from `--skills` paths (supports both single `skill.json` and directories)
2. Load benchmark tasks from `--benchmark` or `--tasks`
3. Build the cartesian product: `skills x tasks x runs`
4. Execute all task items concurrently (controlled by `--concurrency`, with timeout and retry)
5. Score each result using the appropriate scorer
6. Aggregate into `SkillCompletionReport` per skill
7. Write reports to `--output-dir`

### 4. Select Skills

Filter and rank skills based on evaluation reports using a multi-dimensional strategy.

```bash
eval-skills select \
  --from ./skills \
  --reports ./reports/eval-result.json \
  --strategy ./strategy.yaml \
  --min-completion 0.8 \
  --top-k 5 \
  --output ./selected.json
```

| Option | Description | Default |
|--------|-------------|---------|
| `--from <path>` | **Required.** Candidate skills directory or JSON file | — |
| `--reports <file>` | Evaluation reports JSON file | — |
| `--strategy <file>` | SelectStrategy YAML/JSON file | built-in default |
| `--min-completion <rate>` | Override minimum completion rate filter | — |
| `--top-k <n>` | Return only the top K results | all |
| `--output <file>` | Write selected skills to file | stdout |

**Selection pipeline:** Filter (by completion rate, error rate, latency, adapter type, required tags) -> Score -> Rank (by compositeScore, completionRate, latency, or tokenCost) -> TopK

Example `strategy.yaml`:

```yaml
filters:
  minCompletionRate: 0.8
  maxErrorRate: 0.1
  maxLatencyP95Ms: 5000
  adapterTypes: [http, subprocess]
  requiredTags: [production-ready]
sortBy: compositeScore
order: desc
topK: 5
```

### 5. Run Pipeline

Execute the full end-to-end pipeline: **Find -> Eval -> Select -> Report** in a single command.

```bash
eval-skills run \
  --query "math" \
  --benchmark coding-easy \
  --skills-dir ./skills \
  --top-k 3 \
  --min-completion 0.7 \
  --format json markdown \
  --output-dir ./reports
```

This command automates the entire process:
1. **Find** — scans `--skills-dir` and optionally filters by `--query`
2. **Eval** — evaluates all candidate skills against `--benchmark`
3. **Select** — filters and ranks results using `--min-completion`, `--top-k`, and optional `--strategy`
4. **Report** — generates output files in all requested `--format`s

### 6. Generate & Compare Reports

#### Convert report format

```bash
eval-skills report convert \
  --input ./reports/eval-result.json \
  --format html \
  --output ./reports/eval-result.html
```

Supported output formats: `markdown`, `html`.

#### Diff two reports (regression detection)

```bash
eval-skills report diff \
  ./reports/v1.json ./reports/v2.json \
  --label-a "v1.0" --label-b "v2.0" \
  --output ./reports/diff.md
```

Generates a side-by-side delta table per skill showing changes in completion rate, error rate, P95 latency, and composite score with directional arrows.

### 7. Initialize Project

```bash
eval-skills init --dir .
```

Creates the project scaffold:
- `eval-skills.config.yaml` — global configuration
- `skills/` — directory for skill definitions
- `benchmarks/` — directory for benchmark files
- `reports/` — directory for evaluation output

### 8. Manage Configuration

```bash
# List all current configuration values
eval-skills config list

# Get a specific value (supports dot notation)
eval-skills config get llm.model

# Set a value (persisted to ~/.eval-skills/config.yaml)
eval-skills config set concurrency 8
eval-skills config set llm.model gpt-4o
eval-skills config set llm.temperature 0
```

Configuration is resolved in priority order:
1. CLI flags (highest priority)
2. `eval-skills.config.yaml` in current directory
3. `~/.eval-skills/config.yaml`
4. Built-in defaults (`concurrency: 4`, `timeoutMs: 30000`, `outputDir: ./reports`)

## Scorer Types

Each task in a benchmark specifies an evaluator type. The scorer compares the skill's actual output against the expected output.

| Type | Aliases | Description | Score Range |
|------|---------|-------------|-------------|
| `exact_match` | `exact` | Strict equality comparison. Supports `caseSensitive` option. | 0 or 1 |
| `contains` | — | Checks for the presence of all specified keywords in the output. Partial credit: `matched_keywords / total_keywords`. | 0.0 ~ 1.0 |
| `json_schema` | `schema` | Validates output against a JSON Schema (using Ajv). | 0 or 1 |
| `llm_judge` | — | Sends the output + expected rubric to an LLM (configurable model) for quality rating. | 0.0 ~ 1.0 |
| `custom` | — | Loads a custom scorer from `expectedOutput.customScorerPath`. | 0.0 ~ 1.0 |

## Evaluation Metrics

Every evaluation produces a `SkillCompletionReport` with these metrics:

| Metric | Description | Formula |
|--------|-------------|---------|
| **Completion Rate** | Fraction of tasks that passed | `pass_count / total_count` |
| **Partial Score** | Mean score across all tasks | `mean(task_scores)` |
| **Error Rate** | Fraction of tasks that errored or timed out | `(error_count + timeout_count) / total_count` |
| **Consistency Score** | Stability across multiple runs (requires `--runs >= 2`) | `1 - stddev(per_run_completion_rates)` |
| **P50 / P95 / P99 Latency** | Response time percentiles | Sorted percentile of `latencyMs` |
| **Composite Score** | Weighted overall quality score | `0.5 * CR + 0.2 * (1 - latP95_norm) + 0.3 * (1 - ER)` |

## Built-in Benchmarks

| ID | Domain | Tasks | Scoring | Description |
|----|--------|------:|---------|-------------|
| `coding-easy` | coding | 20 | mean / exact_match | Math expressions, string reversal, palindrome detection |
| `skill-quality` | tool-use | 5 | mean / contains | Metadata completeness, description quality, structure checks |
| `web-search-basic` | web | 8 | mean / contains + schema | Factual queries, keyword verification, structured output validation |
| `gaia-v1` | general | — | mean | Placeholder for GAIA benchmark Level 1 tasks |
| `toolbench-lite` | tool-use | — | mean | Placeholder for ToolBench single-tool scenarios |

### Custom Benchmark

Create a `benchmark.json` file:

```json
{
  "id": "my-benchmark",
  "name": "My Custom Benchmark",
  "version": "1.0.0",
  "domain": "general",
  "scoringMethod": "mean",
  "maxLatencyMs": 30000,
  "metadata": { "source": "internal", "lastUpdated": "2026-02-28" },
  "tasks": [
    {
      "id": "task_001",
      "description": "Test basic addition",
      "inputData": { "expression": "2+3" },
      "expectedOutput": { "type": "exact", "value": "5" },
      "evaluator": { "type": "exact" },
      "timeoutMs": 10000,
      "tags": ["math"]
    },
    {
      "id": "task_002",
      "description": "Test keyword presence",
      "inputData": { "query": "TypeScript" },
      "expectedOutput": { "type": "contains", "keywords": ["JavaScript", "Microsoft"] },
      "evaluator": { "type": "contains", "caseSensitive": false },
      "timeoutMs": 15000,
      "tags": ["search"]
    }
  ]
}
```

```bash
eval-skills eval --skills ./my-skill/ --benchmark ./my-benchmark.json
```

## Adapter Types

Skills communicate through adapters. The adapter type is specified in `skill.json` via `adapterType`.

| Adapter | Protocol | How it works | Key config |
|---------|----------|-------------|------------|
| `http` | REST POST | Sends `POST { skillId, version, input }` to `skill.entrypoint`. Supports Bearer / API-Key auth via env vars. | `baseUrl`, `authType`, `authTokenEnvKey` |
| `subprocess` | JSON-RPC 2.0 over stdin/stdout | Spawns `skill.entrypoint` (e.g. `python3 skill.py`), writes JSON-RPC request to stdin, reads response from stdout. | `command`, `args` |
| `mcp` | MCP Protocol | *(Phase 2)* Native Model Context Protocol integration via `@modelcontextprotocol/sdk`. | — |

## Workflow Examples

### Evaluating a Single Skill

```bash
# 1. Create a skill skeleton
eval-skills create --name my_calc --from-template python_script

# 2. Implement your logic in skills/my_calc/skill.py

# 3. Run evaluation against the coding-easy benchmark
eval-skills eval \
  --skills ./skills/my_calc/skill.json \
  --benchmark coding-easy \
  --runs 3 \
  --format json markdown

# 4. Review the report
cat ./reports/eval-result-*.md
```

### Comparing Multiple Candidate Skills

```bash
# 1. Discover candidates
eval-skills find --query "weather" --skills-dir ./skills

# 2. Evaluate all candidates on the same benchmark
eval-skills eval \
  --skills ./skills/weather_v1 ./skills/weather_v2 ./skills/weather_v3 \
  --benchmark web-search-basic \
  --runs 3

# 3. Select the best
eval-skills select \
  --from ./skills \
  --reports ./reports/eval-result-*.json \
  --min-completion 0.8 \
  --top-k 2

# 4. Compare two versions
eval-skills report diff \
  ./reports/v1.json ./reports/v2.json \
  --label-a "weather_v1" --label-b "weather_v2"
```

### Full Pipeline (One Command)

```bash
eval-skills run \
  --skills-dir ./skills \
  --benchmark coding-easy \
  --top-k 3 \
  --min-completion 0.7 \
  --format json markdown html \
  --output-dir ./reports
```

### CI/CD Quality Gate

```bash
# In your CI pipeline — fail the build if completion rate drops below 80%
eval-skills eval \
  --skills ./skills/production_skill \
  --benchmark coding-easy \
  --exit-on-fail \
  --min-completion 0.8 \
  --format json
```

### Regression Detection

```bash
# Compare today's evaluation against the baseline
eval-skills report diff \
  ./reports/baseline.json ./reports/latest.json \
  --label-a "baseline" --label-b "latest" \
  --output ./reports/regression-check.md
```

## Best Practices

1. **Always use `--runs 3` or more** when evaluating for production decisions. Single-run results can be noisy; the consistency score captures stability across runs.

2. **Use `--exit-on-fail` in CI/CD pipelines** to enforce quality gates. Set `--min-completion` to your acceptable threshold (recommended: 0.8 for production skills).

3. **Create domain-specific custom benchmarks** rather than relying solely on built-in ones. Your custom benchmark should reflect real-world inputs your skill will encounter.

4. **Use `report diff` after every skill upgrade** to catch regressions early. Compare the new evaluation against a saved baseline report.

5. **Use `--dry-run` before long evaluations** to validate your configuration (skill paths, benchmark resolution, task count) without actually executing tasks.

6. **Persist results with `--store`** to track skill quality over time. The SQLite store enables historical trend queries.

7. **Start with `--concurrency 1` when debugging** a failing skill, then increase for production benchmarking.

8. **Tag your benchmark tasks** to enable per-category analysis (e.g., filter by `math`, `string`, `edge-case`).

## Skill JSON Schema

Every skill must provide a `skill.json` that conforms to this structure:

```json
{
  "id": "my_skill_v1",
  "name": "My Skill",
  "version": "1.0.0",
  "description": "Does something useful",
  "tags": ["utility", "math"],
  "inputSchema": {
    "type": "object",
    "properties": { "query": { "type": "string" } },
    "required": ["query"]
  },
  "outputSchema": {
    "type": "object",
    "properties": { "result": { "type": "string" } }
  },
  "adapterType": "subprocess",
  "entrypoint": "python3 skill.py",
  "metadata": {
    "author": "Your Name",
    "license": "MIT",
    "homepage": "https://github.com/you/my-skill"
  }
}
```

Validation rules:
- `id`: lowercase alphanumeric with `_` or `-`, non-empty
- `version`: semver format (`X.Y.Z`)
- `adapterType`: one of `http`, `subprocess`, `mcp`, `langchain`, `custom`
- `entrypoint`: non-empty string (URL for http, command for subprocess)

## Global Options

These options are available on all commands:

| Option | Description |
|--------|-------------|
| `-c, --config <path>` | Path to configuration file |
| `--json` | JSON output format (CI-friendly) |
| `--no-color` | Disable colored output |
| `-v, --verbose` | Verbose logging |
| `--version` | Show version |
| `-h, --help` | Show help |