Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →archive228/eval-harness
Build a repeatable eval loop that grades agent output with an LLM judge, so prompt/skill changes get scored against a baseline instead of eyeballed. Reuses loopkit's verifier subagent as the grader — do not build a new one.
Passed 22 of 23 test cases with the skill, and 10 of 23 without it.
and 15 more in the full report
13 fixed10 passed either way1 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 12 test cases the skill fixed. Same model, same prompt, with and without the skill.
Where in the workspace project directory should the verifier agent system prompt file be located according to standard project conventions?
| Checked | Without skill | With skill |
|---|---|---|
| Identifies .claude/agents/verifier.md as the verifier agent file path | Missed | Passed |
The instructions your model reads when the skill loads.
Every prompt tweak in a long-running agent looks like an improvement in the moment. The only way to know is a graded run against fixed inputs. Loopkit already ships .claude/agents/verifier.md — that is your grader. Do not rebuild it.
inputs.jsonl → runner → outputs.jsonl → verifier (per row) → verdicts.jsonl → diff vs baseline
Each stage writes to disk. No stage holds the whole run in context.
One JSON object per row: {"id": "case-01", "input": "...", "expected": "..."}.
inputs-v2.jsonl) when you change it. Never edit in place — you lose the baseline.A dumb loop: for each row, call the model with the current prompt/skill, capture output, write {"id": ..., "output": ...} to outputs.jsonl. No grading here — just capture.
If the runner is smart it will bias the eval. Keep it dumb.
Fan out one subagent per row (see subagent-fanout). Each gets:
.claude/agents/verifier.md.Verifier returns strict JSON: {"pass": bool, "why": "..."}. Collect into verdicts.jsonl.
test cases passed, vs 10 / 23 without
More skills from the same source and nearby in the registry, with their headline benchmark lift.