Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →openhands/manage-evals
This skill should be used when the user asks to "trigger an eval", "run evaluation", "run swebench", "run gaia", "run benchmark", "compare eval runs", "compare evaluation results", "check eval regression", "compare benchmark results", "what changed in the eval", "diff eval runs", or mentions triggering, comparing, or reporting on SWE-bench, GAIA, or other benchmark evaluation results. Provides workflow for triggering evaluations on different benchmarks, finding and comparing runs, and reporting
Passed 15 of 19 test cases with the skill, and 5 of 19 without it.
and 13 more in the full report
15 fixed4 passed either way3 still failing2 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 15 test cases the skill fixed. Same model, same prompt, with and without the skill.
I need to run an evaluation on the commit SHA 'a1b2c3d' for SWE-bench Pro using 200 instances. I want to test the 'planning' tool preset and the 'acp-claude' agent type. What command should I execute?
| Checked | Without skill | With skill |
|---|---|---|
| Executes python .agents/skills/manage-evals/scripts/manage_evals.py trigger command. | Missed | Passed |
| Includes --tool-preset planning argument. | Passed | Passed |
| Includes --agent-type acp-claude argument. | Passed | Passed |
The instructions your model reads when the skill loads.
OpenHands evaluations produce results stored on a CDN at https://results.eval.all-hands.dev/. Each run is identified by a path: {benchmark}/{model_slug}/{github_run_id}/. This skill enables triggering evaluation runs, comparing results between runs, and posting performance reports as GitHub PR comments.
python .agents/skills/manage-evals/scripts/manage_evals.py trigger \
--sdk-ref <BRANCH_OR_TAG> --benchmark swebench --eval-limit 50
python .agents/skills/manage-evals/scripts/manage_evals.py compare \
"<benchmark>/<model_slug>/<run_id>/" \
--auto-baseline
python .agents/skills/manage-evals/scripts/manage_evals.py compare \
"<benchmark>/<model_slug>/<run_id>/" \
--auto-baseline \
--post-comment --pr <PR_NUMBER> --repo OpenHands/software-agent-sdk
# SWE-bench (default) on a PR branch
python .agents/skills/manage-evals/scripts/manage_evals.py trigger \
--sdk-ref my-feature-branch --eval-limit 50
test cases passed, vs 5 / 19 without
More skills from the same source and nearby in the registry, with their headline benchmark lift.