Agent Harness

Agent Harness

Most agents ship on vibes: someone tries eight prompts, the output looks good, it goes to production, and the next prompt tweak silently breaks a refusal nobody re-tested. This skill builds the harness around an agent so its behaviour becomes measurable — scenario suites with structural assertions, deterministic replay of recorded tool calls, paired regression diffing across prompt and model changes, and per-scenario cost and latency budgets. The tools here score an agent; they never invoke one, so they run offline on every commit.

When to use this skill

• An agent is going to production and the only quality evidence is manual spot-checking
• A prompt, tool schema, or model version is changing and you need to know what broke
• Two model or configuration options need a defensible comparison, not a demo
• An incident happened and you need the behaviour encoded as a permanent regression test
• Agent cost or latency is climbing across releases and nobody can point to when
• An existing eval suite reports a healthy pass rate that nobody trusts

Inputs the skill expects

• The agent's tool inventory — names, arguments, and which tools are irreversible
• Recorded transcripts per scenario: tool calls, final output, turns, latency, cost, error state
• The behavioural rules the agent must hold (refusals, escalation triggers, policy boundaries)
• Known failure history — past incidents, customer complaints, internal bug reports
• Current cost and latency expectations per interaction
• The release gate that consumes the result (CI job, review checklist, launch review)

Clarify First

Before building the harness, confirm these inputs. If any is unknown or vague, ASK — do not assume:

• Which agent actions are irreversible — determines which scenarios need tool_not_called assertions at critical severity, and what the release gate blocks on
• Whether transcripts are already recorded — decides whether workflow 1 starts from replay or from an instrumentation task first
• What the suite gates — a CI blocking check, a nightly report, or a one-off comparison; changes suite size, runtime budget, and severity strictness
• The known failure modes — past incidents seed the adversarial and refusal buckets, which is where regressions actually hide

Stop rule: ask only the 2-3 that most change the output. If the user says "just draft it," proceed and list your assumptions at the top of the artifact.

Workflows