Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →sonichi/voice-agent-test-harness
Drive a fixed suite of spoken tests against a voice agent ("subject") from a co-located machine ("prober"), measure response latency / clarity / accuracy, diff against baseline, and report to the owner over Telegram.
Passed 18 of 22 test cases with the skill, and 5 of 22 without it.
and 18 more in the full report
14 fixed5 passed either way4 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 14 test cases the skill fixed. Same model, same prompt, with and without the skill.
Which command-line utility and driver framework does the prober machine use on macOS to capture ambient audio from the voice subject?
| Checked | Without skill | With skill |
|---|---|---|
| Identifies sox rec as the recording utility. | Missed | Passed |
| Names CoreAudio as the macOS audio framework. | Passed | Passed |
The instructions your model reads when the skill loads.
Drive a fixed suite of spoken tests against a voice agent ("subject") from a co-located machine ("prober"), measure response latency / clarity / accuracy, diff against baseline, and report to the owner over Telegram.
Design: docs/voice-agent-test-framework.md
v1 (macOS). Real audio path: TTS via
gemini-tts+afplay, mic capture viasoxrec(CoreAudio), voice-onset via numpy RMS, STT + judge via Gemini (Sutando-standard,GEMINI_API_KEY). Manual trigger; reports to owner only. Each prober-side component is tested; the full closed loop needs the second laptop speaking.
So that a half-SKIPPED suite is never mistaken for "mostly fine," here is exactly what executes through the real acoustic path now versus what is stubbed or excluded. A captured live run is committed at examples/run-2026-06-06.json.
| Capability | Status today |
|---|---|
Single-answer suite (test_cases.yaml, core-v1) — speak → capture → onset → Gemini STT → judge → score | ✅ Wired. Every row runs end-to-end on real audio; pass / fail / partial / no_response are all measured outcomes, not stubs. |
| Latency / clarity / accuracy scoring + baseline diff + Telegram roll-up | ✅ Wired — computed on real captured turns. |
timer action test — real side-effect verify (waits, listens for the alarm) | ✅ Wired. |
Multi-turn workflow turns (workflow_cases.yaml, e.g. the developer code-change flow) | ⚠️ Partial. The spoken handling is captured and judged; remote side effects (branch/test/cleanup) are not observable from the prober, so these score wording only. |
| Gmail / CRM workflow turns | ⛔ Excluded — unfinished test setup; omitted from results, not reported as failures. |
| Daily auto-scheduling | ⛔ Not wired — manual trigger only. |
cd ~/GitHub/sutando/skills/voice-agent-test-harness
python3 scripts/run_suite.py --quick # --quick shortens the 2-min timer wait to 30s
--deliver to send the report to your Telegram.Useful flags:
python3 scripts/run_suite.py --only arithmetic # one test by id
python3 scripts/run_suite.py --dry-run # no audio/model; canned data (CI/sanity)
python3 scripts/baseline.py --promote results/voice-test/<date>.json # set regression baseline
test cases passed, vs 5 / 22 without
More skills from the same source and nearby in the registry, with their headline benchmark lift.