Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →perminder-klair/subwave-llm-bench
Benchmark and compare LLM models for SUB/WAVE's on-air calls — track picks, segments, listener requests, DJ scripts, banter, and programme beats — in both candidate-pool and agent modes, using controller/scripts/llm-bench (the matrix harness) or the legacy picker-only controller/scripts/picker-test.mjs.
Passed 18 of 21 test cases with the skill, and 6 of 21 without it.
and 16 more in the full report
16 fixed5 passed either way1 still failing2 broke
Tokens, turns and time describe resource use, not a dollar price.
This benchmark measured an earlier version of the skill.
A newer attempt on gemini-3.8-flash on 10/21/2026 did not produce a publishable comparison (not comparable). The snapshot above is the previously measured run.
One of 16 test cases the skill fixed. Same model, same prompt, with and without the skill.
Where does llm-bench write its diffable JSON benchmark reports by default when no --out path is specified?
| Checked | Without skill | With skill |
|---|---|---|
| Identifies scripts/llm-bench/reports as the default benchmark report output directory. | Missed | Passed |
The instructions your model reads when the skill loads.
Two harnesses, one job: measure how well a provider + model handles the station's real LLM calls before trusting it on air.
llm-bench (controller/scripts/llm-bench/, npm run llm-bench) — the
primary harness. A matrix runner over every on-air call kind: pool picks
(pickNextTrack), agent picks (djAgentPick), Track Shortlist picks
(djShortlistPick), pool + agent segments
(generateSegment / djAgentSegment), request matching (matchRequest /
djAgentRequest), the free-text scripts (intro, link, station ID, hourly),
banter, and the programme family (plan, beats, exchanges). Scores
reliability + deterministic rule checks, prints a per-model comparison
table, writes a diffable JSON report.picker-test.mjs — the legacy picker-only deep-dive. Still useful for
high-iteration picker runs comparable with historical results, and it has a
bundled orchestration script (see the legacy section at the end).Both import the live prompts and schemas from src/ (never copies), fake
everything external (library, tools, weather/news data), and override
provider/model only inside their own short-lived process — the live
controller's configured model is never touched.
@ai-sdk/* provider
translates tools / structured output differently. Canonical case:
deepseek-v4-flash scored 0/4 via the deepseek direct provider but
4/4 via openrouter. Always benchmark through the routing you'll
actually deploy — "is model X good?" is the wrong question; "is
provider+X good?" is the right one.djAgentPick/long-context (full prompt +
tool loop), generateSegment/dull-weather (the model must decline to air
— small models botch the silence encoding; a huge wall-clock here means the
structured-output retry rescued a failed first attempt), and the
multi-voice JSON kinds (generateBanter, generateProgrammeExchange).test cases passed, vs 6 / 21 without
More skills from the same source and nearby in the registry, with their headline benchmark lift.