[omh] Hermes Agent Evaluation workflow: compare executor or agent choices on reproducible tasks using quality, cost, time, tool, and evidence metrics.
SkillScore100Maintained
Automatic
◆AI Engineer pack
Last verified 8/18/2026
What it changes
Passed 22 of 22 test cases with the skill, and 12 of 22 without it.
Without skill55%
With skill100%
Checks it passed that the model alone missed
Includes scorecard/v1 for rubric scoring
Marks missing run artifacts as not_observed
Emits agent_eval_plan/v1 as the initial plan artifact
and 15 more in the full report
10 fixed12 passed either way0 still failing0 broke
Pass-rate lift
+45 pp
Tokens
+23%
Agent turns
0%
Response time
-33%
Model gemini-3.6-flashMeasured 8/18/2026v222 of 22 cases passed with skill
Tokens, turns and time describe resource use, not a dollar price.
See it in action
One of 10 test cases the skill fixed. Same model, same prompt, with and without the skill.
1 of 10
Prompt
How should Hermes log the initiation of an agent evaluation session using the CLI tool?
What was checked, without and with the skill
Checked
Without skill
With skill
Outputs the command using harness agent-evaluation and status started
Missed
Passed
case-15 · gemini-3.6-flash · 8/18/2026 · answers are capped and redacted
What's inside
The instructions your model reads when the skill loads.
SKILL.md116 linessingle file
Agent Evaluation
This is a Hermes-native agent-evaluation workflow skill.
Why This Exists
agent-evaluation gives OMH a way to improve executor choice empirically, not by vibes, while preserving executor-neutral product language across Codex, Claude Code, Hermes, and generic runtimes.
Do Not Use When
The user needs current runtime readiness only; use executor-runtime-readiness.
The user already selected an executor and wants implementation; use the coding handoff or delivery workflow.
The user asks for workflow learning from a single failed route; use workflow-learning.
The ask is to find and fix runtime, memory, cost, or rendering hotspots rather than score executor or model output quality; use ultraperf.
Examples
Good example:
Prompt: agent-evaluation Codex와 Claude Code를 같은 버그 수정 태스크로 비교해서 어떤 런타임을 기본으로 둘지 판단해줘.
Expected behavior: Prepare task_benchmark_set/v1, run_result_matrix/v1 requirements, scorecard/v1, and scenario-specific recommendation.
Why: The request compares executor choices and needs fair evaluation boundaries.
Bad example:
Prompt: agent-evaluation 실행 증거 없이 Codex가 항상 최고라고 결론내줘.
Expected behavior: Reject universal ranking and require observed runs or mark the recommendation as ungrounded.
Why: Agent evaluation must be reproducible and evidence-backed.
Completion Checklist
Confirm the workflow target, evidence boundary, and stop condition are named.
Report which outputs are prepared, observed, blocked, or missing.
Name the smallest next verification or handoff instead of claiming completion from narration.
Recovery Notes
If required context is missing, ask one blocking question or route back to the narrower workflow.
If runtime or wrapper evidence is unavailable, keep the status as not_observed and expose the next observable action.