---
name: mgechev/skillgrade-setup
source: https://app.decimal.ai/s/mgechev-skillgrade-setup@1/SKILL.md
source_sha256: 0d139604f5ce
---

# Skillgrade Evaluation Setup

## Procedures

**Step 1: Install Skillgrade**
1. Verify Node.js 20+ and Docker are available.
2. Run `npm i -g skillgrade` to install the CLI globally.

**Step 2: Initialize an Eval Configuration**
1. Navigate to the skill directory (must contain a `SKILL.md`).
2. Set the appropriate API key environment variable (`GEMINI_API_KEY`, `ANTHROPIC_API_KEY`, or `OPENAI_API_KEY`).
3. Run `skillgrade init` to generate an `eval.yaml` with AI-powered tasks and graders.
4. If an `eval.yaml` already exists, pass `--force` to overwrite: `skillgrade init --force`.
5. Without an API key, a well-commented template is generated instead.

**Step 3: Configure eval.yaml**
1. Read `references/eval-yaml-spec.md` for the full configuration schema.
2. Define one or more tasks under the `tasks:` key. Each task requires:
   - `name`: unique task identifier
   - `instruction`: what the agent should accomplish
   - `workspace`: files to copy into the evaluation container
   - `graders`: one or more scoring mechanisms (see the `skillgrade-graders` skill)
3. Optionally configure `defaults:` for agent, provider, trials, timeout, and threshold.

**Step 4: Run Evaluations**
1. Select an appropriate preset based on the evaluation goal:
   - `--smoke` (5 trials): Quick capability check.
   - `--reliable` (15 trials): Reliable pass rate estimate.
   - `--regression` (30 trials): High-confidence regression detection.
2. Run the evaluation: `skillgrade --smoke`.
3. Run a specific eval by name: `skillgrade --eval=fix-linting`.
4. Run multiple evals: `skillgrade --eval=fix-linting,write-tests`.
5. Run only deterministic graders (skip LLM calls): `skillgrade --grader=deterministic`.
6. Run only LLM rubric graders: `skillgrade --grader=llm_rubric`.
7. The agent is auto-detected from the API key. Override with `--agent=gemini|claude|codex|acp|opencode|command`.
8. For ACP, pass `--acp-command="gemini --acp"` or set `defaults.acp.command`.
9. For OpenCode, pass `--opencode-agent=build|plan|explore` or `--opencode-model=provider/model`.
10. For a custom agent, pass `--agent=command --command="node mycli.js"` or set `defaults.command`. The instruction is piped to the command's stdin.
11. Override the provider with `--provider=docker|local`.

**Step 5: Review Results**
1. Run `skillgrade preview` for a CLI report.
2. Run `skillgrade preview browser` to open the web UI at `http://localhost:3847`.
3. Reports are saved to `$TMPDIR/skillgrade/<skill-name>/results/`. Override with `--output=DIR`.

**Step 6: Integrate with CI**
1. Add a GitHub Actions step that installs skillgrade, navigates to the skill directory, and runs with `--regression --ci --provider=local`.
2. Use `--provider=local` in CI — the runner is already an ephemeral sandbox, so Docker adds overhead without benefit.
3. The `--ci` flag causes a non-zero exit code if the pass rate falls below `--threshold` (default: 0.8).
4. Read `references/ci-example.md` for a complete workflow template.

## Error Handling
* If `skillgrade init` fails with "No SKILL.md found," verify the current directory contains a valid `SKILL.md` file.
* If evaluation hangs, check Docker is running and the container has network access for API calls.
* If all trials fail with "No API key," ensure the environment variable is exported, not just set inline for a different command.