Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →Search phone-call history for when a feature regressed (find-regression.py) and drill into a single call to see what went wrong (diagnose-call.py). Skips reading 100+ transcripts by hand.
.claude/skills/sonichi-regression-search/SKILL.md| Test case | Without → With | Effect | Δ tokens | Δ turns |
|---|---|---|---|---|
| case-02 | ✗→✓ | ▲ Improved | 236% | 0% |
| case-13 | ✗→✓ | ▲ Improved | -24% | 0% |
| case-14 | ✗→✓ | ▲ Improved | -24% | 0% |
| case-15 | ✗→✓ | ▲ Improved | -37% | 0% |
| case-16 | ✗→✓ | ▲ Improved | -34% | 0% |
Two scripts for hunting down bad calls without reading every transcript:
find-regression.py — search results/calls/calls.jsonl for calls touching a feature, classify each as working/broken, print a sorted timeline.diagnose-call.py — drill into a single call by SID, report refusals/errors/silences/repeated requests, optionally show metrics from data/call-metrics.jsonl.Closes #188.
bashpython3 skills/regression-search/scripts/find-regression.py "record" python3 skills/regression-search/scripts/find-regression.py "summon" --since 2026-04-01 python3 skills/regression-search/scripts/find-regression.py "play" --json
Flags:
--since YYYY-MM-DD — only show calls on/after this date--json — machine-readable output--show-snippet — print a one-line transcript snippet for each callA call is broken for a query if any of:
Otherwise the call is working if Sutando's response includes the feature keyword and isn't flagged broken.
These are intentionally crude — the goal is "good enough to find the regression window without reading 163 transcripts." Tune as you find false positives.
record. The issue calls this out as future work.bashpython3 skills/regression-search/scripts/diagnose-call.py de1f04733fc2 python3 skills/regression-search/scripts/diagnose-call.py CA701fc4129779... --metrics python3 skills/regression-search/scripts/diagnose-call.py de1f04733fc2 --json
Accepts a full SID or just the last 12 characters. Reports turn counts, refusals, errors, silences, repeated user requests, and the ending style (normal vs abrupt user end vs sutando silence). With --metrics, also pulls per-event tool-call timeline from data/call-metrics.jsonl (requires PR #223). Exit code 1 if any issues are found, 0 if clean — useful for CI.
Typical workflow: run find-regression.py to surface broken candidates, then diagnose-call.py <sid> to drill into the worst one.
| Case | Status | Duration (ms) | Turns | Tokens | Tool calls | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Without | With | Δ | Without | With | Δ | Without | With | Δ | Without | With | Δ | ||
case-05 | fail→fail | 17,864 | 18,234 | +2% | 1 | 1 | 0% | 2,898 | 3,810 | +31% | 0 | 0 | — |
case-02 | fail→pass | 7,204 | 17,456 | +142% | 1 | 1 | 0% | 1,119 | 3,758 | +236% | 0 | 0 | — |
case-03 | fail→fail | 2,836 | 11,063 | +290% | 1 | 1 | 0% | 340 | 1,215 | +257% | 0 | 0 | — |
case-04 | pass→pass | 8,347 | 10,540 | +26% | 1 | 1 | 0% | 1,498 | 2,527 | +69% | 0 | 0 | — |
case-01 | fail→fail | 10,751 | 13,665 | +27% | 1 | 1 | 0% | 1,934 | 1,119 | -42% | 0 | 0 | — |
case-06 | pass→pass | 18,195 | 19,879 | +9% | 1 | 1 | 0% | 3,216 | 4,440 | +38% | 0 | 0 | — |
case-07 | pass→pass | 13,751 | 3,358 | -76% | 1 | 1 | 0% | 2,132 | 1,355 | -36% | 0 | 0 | — |
case-08 | pass→pass | 16,205 | 3,384 | -79% | 1 | 1 | 0% | 2,508 | 1,387 | -45% | 0 | 0 | — |
case-09 | pass→pass | 10,442 | 1,822 | -83% | 1 | 1 | 0% | 1,485 | 1,132 | -24% | 0 | 0 | — |
case-10 | pass→pass | 12,700 | 2,694 | -79% | 1 | 1 | 0% | 1,815 | 1,163 | -36% | 0 | 0 | — |
case-11 | pass→pass | 14,976 | 2,903 | -81% | 1 | 1 | 0% | 2,274 | 1,294 | -43% | 0 | 0 | — |
case-12 | pass→pass | 10,707 | 2,129 | -80% | 1 | 1 | 0% | 1,465 | 1,116 | -24% | 0 | 0 | — |
case-13 | fail→pass | 8,724 | 1,555 | -82% | 1 | 1 | 0% | 1,419 | 1,072 | -24% | 0 | 0 | — |
case-14 | fail→pass | 9,352 | 1,774 | -81% | 1 | 1 | 0% | 1,358 | 1,031 | -24% | 0 | 0 | — |
case-15 | fail→pass | 11,029 | 1,667 | -85% | 1 | 1 | 0% | 1,659 | 1,041 | -37% | 0 | 0 | — |
case-16 | fail→pass | 12,108 | 2,721 | -78% | 1 | 1 | 0% | 1,989 | 1,305 | -34% | 0 | 0 | — |
case-17 | pass→pass | 12,245 | 1,690 | -86% | 1 | 1 | 0% | 1,917 | 1,080 | -44% | 0 | 0 | — |
case-18 | pass→pass | 10,809 | 4,551 | -58% | 1 | 1 | 0% | 1,573 | 1,513 | -4% | 0 | 0 | — |
case-19 | fail→pass | 18,466 | 6,784 | -63% | 1 | 1 | 0% | 2,838 | 2,078 | -27% | 0 | 0 | — |
case-20 | fail→pass | 8,396 | 2,183 | -74% | 1 | 1 | 0% | 1,346 | 1,193 | -11% | 0 | 0 | — |
case-21 | fail→fail | 10,354 | 3,835 | -63% | 1 | 1 | 0% | 1,674 | 1,551 | -7% | 0 | 0 | — |
case-22 | fail→pass | 11,478 | 1,589 | -86% | 1 | 1 | 0% | 1,743 | 1,048 | -40% | 0 | 0 | — |
DecimalAI ran this skill against gemini-3.6-flash twice over the same eval suite — once with the skill loaded and once without — and compared the two runs case by case. 22 cases were attempted, and 20 counted toward the lift figure. The other 2 produced results that are not comparable between the two arms, so they are excluded from the headline rather than averaged into it. The headline lift of +36 percentage points is the difference between those two pass rates over the 20 comparable cases.
Without the skill loaded, the model failed this case. With it loaded, the same prompt on the same model passed. This is one improved case from the latest verified run; every case, including any that regressed, is in the table above.
Other measured skills in the registry, with their headline benchmark lift.