Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →Run the corpus benchmark — booster locally, optional Gemini/Sonnet/Opus baselines — and persist a verifiable measured-vs-claimed table
.claude/skills/ruvnet-cost-benchmark/SKILL.md| Test case | Without → With | Effect | Δ tokens | Δ turns |
|---|---|---|---|---|
| case-11 | ✗→✓ | ▲ Improved | -3% | 0% |
| case-01 | ✗→✓ | ▲ Improved | 247% | 0% |
| case-06 | ✗→✓ | ▲ Improved | 1% | 0% |
| case-02 | ✗→✓ | ▲ Improved | -13% | 0% |
| case-04 | ✗→✓ | ▲ Improved | -29% | 0% |
Runs scripts/bench.mjs against the structural+adversarial corpus and writes per-case + summary results to docs/benchmarks/runs/. This is the verification gate that backs every measurable claim in cost-booster-edit / cost-booster-route.
bench/booster-corpus.json — confirm new cases route correctly.BENCH_ANTHROPIC=1.v3/ (where agent-booster resolves):bash ( cd v3 && node ../plugins/ruflo-cost-tracker/scripts/bench.mjs ) # booster only — free, ~85 ms ( cd v3 && BENCH_LLM_BASELINE=1 node ../plugins/ruflo-cost-tracker/scripts/bench.mjs ) # + Gemini 2.0 Flash (cheap) ( cd v3 && BENCH_LLM_BASELINE=1 BENCH_ANTHROPIC=1 \ node ../plugins/ruflo-cost-tracker/scripts/bench.mjs ) # + Sonnet 4.6 + Opus 4.7
winRate (Tier 1 cases). Adversarial cases are tracked separately as escalationRate.docs/benchmarks/runs/latest.json — pointer to the most recent rundocs/benchmarks/runs/<ISO-timestamp>.json — historical recordcost-report step 2 reads latest.json for live tier-spend numbers).winRate ≥ 0.80 on Tier 1 cases (smoke step 23). Lower the threshold by editing scripts/smoke.sh.escalationRate is reported but ungated — adversarial cases are diagnostic.| Env var | Default | Purpose | |---|---|---| | BENCH_LLM_BASELINE | unset | =1 runs the OpenAI-compat baseline | | BENCH_LLM_MODEL | models/gemini-2.0-flash | Override the OpenAI-compat model | | BENCH_LLM_BASE_URL | Gemini OpenAI shim | Override endpoint | | BENCH_ANTHROPIC | unset | =1 runs Anthropic baseline (Sonnet 4.6 + Opus 4.7) | | BENCH_ANTHROPIC_MODELS | claude-sonnet-4-6,claude-opus-4-7 | Comma-separated Claude IDs | | BENCH_OUT | timestamped file | Override output path | | BENCH_QUIET=1 | unset | Suppress markdown summary |
API keys auto-pulled from gcloud secrets (GOOGLE_AI_API_KEY, ANTHROPIC_API_KEY); override with BENCH_LLM_API_KEY / BENCH_ANTHROPIC_API_KEY.
ADR-0002 §"Decision 1" / §"Riskiest assumption" · cost-booster-edit/SKILL.md (verification table consumes this skill's output) · cost-report/SKILL.md step 2 (reads runs/latest.json).
| Case | Status | Duration (ms) | Turns | Tokens | Tool calls | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Without | With | Δ | Without | With | Δ | Without | With | Δ | Without | With | Δ | ||
case-11 | fail→pass | 6,112 | 3,431 | -44% | 1 | 1 | 0% | 1,227 | 1,194 | -3% | 0 | 0 | — |
case-01 | fail→pass | 2,596 | 3,938 | +52% | 1 | 1 | 0% | 323 | 1,120 | +247% | 0 | 0 | — |
case-06 | fail→pass | 7,592 | 2,929 | -61% | 1 | 1 | 0% | 1,468 | 1,486 | +1% | 0 | 0 | — |
case-02 | fail→pass | 8,002 | 6,056 | -24% | 1 | 1 | 0% | 1,451 | 1,262 | -13% | 0 | 0 | — |
case-03 | fail→fail | 13,539 | 4,107 | -70% | 1 | 1 | 0% | 2,193 | 1,073 | -51% | 0 | 0 | — |
case-04 | fail→pass | 18,101 | 1,458 | -92% | 1 | 1 | 0% | 1,572 | 1,117 | -29% | 0 | 0 | — |
case-05 | fail→pass | 30,019 | 3,119 | -90% | 1 | 1 | 0% | 1,672 | 1,468 | -12% | 0 | 0 | — |
case-07 | fail→pass | 3,003 | 1,549 | -48% | 1 | 1 | 0% | 575 | 1,166 | +103% | 0 | 0 | — |
case-08 | fail→pass | 11,326 | 1,341 | -88% | 1 | 1 | 0% | 2,094 | 1,074 | -49% | 0 | 0 | — |
case-09 | fail→pass | 17,796 | 1,500 | -92% | 1 | 1 | 0% | 3,341 | 1,116 | -67% | 0 | 0 | — |
case-10 | fail→pass | 11,966 | 2,647 | -78% | 1 | 1 | 0% | 2,673 | 1,435 | -46% | 0 | 0 | — |
case-12 | fail→pass | 14,819 | 1,514 | -90% | 1 | 1 | 0% | 747 | 1,064 | +42% | 0 | 0 | — |
case-13 | fail→pass | 8,138 | 2,023 | -75% | 1 | 1 | 0% | 1,332 | 1,167 | -12% | 0 | 0 | — |
case-14 | fail→pass | 9,598 | 3,257 | -66% | 1 | 1 | 0% | 1,632 | 1,508 | -8% | 0 | 0 | — |
case-15 | fail→pass | 8,251 | 2,002 | -76% | 1 | 1 | 0% | 1,321 | 1,196 | -9% | 0 | 0 | — |
case-16 | fail→pass | 10,324 | 1,242 | -88% | 1 | 1 | 0% | 1,886 | 1,088 | -42% | 0 | 0 | — |
case-17 | fail→pass | 11,726 | 2,397 | -80% | 1 | 1 | 0% | 1,963 | 1,298 | -34% | 0 | 0 | — |
case-18 | fail→pass | 11,255 | 1,560 | -86% | 1 | 1 | 0% | 1,806 | 1,106 | -39% | 0 | 0 | — |
case-19 | pass→pass | 7,337 | 3,294 | -55% | 1 | 1 | 0% | 1,241 | 1,138 | -8% | 0 | 0 | — |
case-20 | pass→pass | 2,528 | 2,472 | -2% | 1 | 1 | 0% | 414 | 1,297 | +213% | 0 | 0 | — |
case-21 | pass→pass | 13,827 | 6,345 | -54% | 1 | 1 | 0% | 2,793 | 2,077 | -26% | 0 | 0 | — |
case-22 | fail→fail | 4,377 | 3,576 | -18% | 1 | 1 | 0% | 794 | 1,311 | +65% | 0 | 0 | — |
DecimalAI ran this skill against gemini-3.6-flash twice over the same eval suite — once with the skill loaded and once without — and compared the two runs case by case. 22 cases were attempted, and 20 counted toward the lift figure. The other 2 produced results that are not comparable between the two arms, so they are excluded from the headline rather than averaged into it. The headline lift of +77 percentage points is the difference between those two pass rates over the 20 comparable cases.
Without the skill loaded, the model failed this case. With it loaded, the same prompt on the same model passed. This is one improved case from the latest verified run; every case, including any that regressed, is in the table above.
Other measured skills in the registry, with their headline benchmark lift.