Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →hashgraph-online/eval-regression
Use when the user asks to check a plugin or skill with behavioral evals, compare it with HEAD, or investigate a behavioral regression.
Passed 14 of 19 test cases with the skill, and 3 of 19 without it.
and 15 more in the full report
15 fixed3 passed either way5 still failing1 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 15 test cases the skill fixed. Same model, same prompt, with and without the skill.
What is the maximum number of parallel sessions allowed during a standard candidate evaluation run before needing explicit approval?
| Checked | Without skill | With skill |
|---|---|---|
| Identifies 4 as the maximum session limit for standard evaluation runs. | Missed | Passed |
The instructions your model reads when the skill loads.
Use deterministic repository tests first. Stop when the target has no behavioral diff.
Resolve the plugin from the argument or cwd. Its catalog is evals/evals.json; the shared runner is ../ai-agent-bench/scripts/run_evals.py. Require the user to choose agent, model, and effort. Never select a costly model or high effort silently.
--changed-from <base>, or name them with --case. Do not select the full catalog implicitly.--run. The runner prints the cases, modes, repeat count, session count, per-session timeout, and maximum duration without starting an agent.--run. The default is candidate-only, one run per case, 180 seconds per session, and at most four sessions.git archive; run the same selected cases, agent, model, effort, repeat, and timeout on both; compare reports with --compare.--all, --repeat > 1, a larger --max-sessions, or a longer timeout needs a new run plan and explicit approval.Remove temporary base copies after the comparison. Do not edit or commit the target.
Deterministic assertions are tool, tool_not, clean_worktree, changed_files_exact, file_contains, file_not_contains, transcript_contains, and tool_sequence. Use a semantic judge only when no filesystem, command, tool, ordering, or assistant-output observation can express the contract.
test cases passed, vs 3 / 19 without
More skills from the same source and nearby in the registry, with their headline benchmark lift.