Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

Regression Search · sonichi | DecimalAI

testingby sonichi

Regression Search

sonichi/regression-search

Source sonichi/sutando ↗License MIT

Search phone-call history for when a feature regressed (find-regression.py) and drill into a single call to see what went wrong (diagnose-call.py). Skips reading 100+ transcripts by hand.

SkillScore82Maintained

Automatic

QA & Test Engineering pack

Last verified 8/23/2026

What it changes

Passed 16 of 20 test cases with the skill, and 9 of 20 without it.

Without skill45%

With skill82%

Checks it passed that the model alone missed

Includes the --metrics flag
Recommends passing the --json flag to find-regression.py
Identifies results/calls/calls.jsonl as the input log path

and 7 more in the full report

8 fixed10 passed either way4 still failing0 broke

Pass-rate lift: +36 pp
Tokens: -8%
Agent turns: 0%
Response time: -54%

Model gemini-3.6-flashMeasured 8/23/2026v118 of 22 cases passed with skill · 20 comparable

Tokens, turns and time describe resource use, not a dollar price.

Use this skill

Save it into your project's skills folder. Claude Code picks it up in its next session.

curl -fsSL --create-dirs -o .claude/skills/sonichi-regression-search/SKILL.md https://app.decimal.ai/s/sonichi-regression-search/SKILL.md

Want it pinned and updatable? pip install "decimalai>=0.15", then decimalai skills add sonichi-regression-search, and start Claude Code with claude --plugin-dir "$(git rev-parse --show-toplevel)/.agents"

1 · One-time — install the CLI (docs ↗)

pip install "decimalai>=0.15"

2 · Get the skill — writes .agents/skills/sonichi-regression-search/ and pins it in .decimal/skills.lock

decimalai skills add sonichi-regression-search

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/sonichi-regression-search/SKILL.md https://app.decimal.ai/s/sonichi-regression-search/SKILL.md

Raw file: /s/sonichi-regression-search/SKILL.md ↗ · machine-readable summary at /s/sonichi-regression-search.json ↗

Safety checks: 3 of 3 passed

Code scan CleanIntent PassedContent Clean

See it in action

One of 8 test cases the skill fixed. Same model, same prompt, with and without the skill.

1 of 8

Prompt

When running call diagnostics with the metrics flag enabled, from which file are event tool-call timelines loaded?

What was checked, without and with the skill
Checked	Without skill	With skill
Identifies data/call-metrics.jsonl as the metrics data path	Missed	Passed

case-15 · gemini-3.6-flash · 8/23/2026 · answers are capped and redacted

What's inside

The instructions your model reads when the skill loads.

SKILL.md62 lines3 filesincludes scripts

Regression Search

Two scripts for hunting down bad calls without reading every transcript:

find-regression.py — search results/calls/calls.jsonl for calls touching a feature, classify each as working/broken, print a sorted timeline.
diagnose-call.py — drill into a single call by SID, report refusals/errors/silences/repeated requests, optionally show metrics from data/call-metrics.jsonl.

Closes #188.

When to use

"When did the X feature stop working?" — pass the feature keyword.
"Has feature Y improved?" — see the broken/working trend over time.
Before shipping a fix — sanity check that the regression is reproducible.

Usage

python3 skills/regression-search/scripts/find-regression.py "record"
python3 skills/regression-search/scripts/find-regression.py "summon" --since 2026-04-01
python3 skills/regression-search/scripts/find-regression.py "play" --json

Flags:

--since YYYY-MM-DD — only show calls on/after this date
--json — machine-readable output
--show-snippet — print a one-line transcript snippet for each call

Heuristics

A call is broken for a query if any of:

Sutando refuses ("I can't", "I'm not able", "I'm unable", "sorry I cannot")
Sutando reports an error ("error", "failed", "didn't work", "something went wrong")
The user repeats the same request 2+ times in a row (Sutando didn't respond usefully)
Sutando says "(Silence)" after the user mentions the feature

Otherwise the call is working if Sutando's response includes the feature keyword and isn't flagged broken.

These are intentionally crude — the goal is "good enough to find the regression window without reading 163 transcripts." Tune as you find false positives.

Track record

Benchmark

16 / 20

test cases passed, vs 9 / 20 without

Adoption

Users: 0
Times used: 0

Safety and source details

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v4 — 8/3/2026

View 15 scanner checks

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

sonichi / sutando

skills/regression-search/SKILL.md

View on GitHub ↗

License MITImported 8/3/2026Version v1Bundled 2 files · scripts

More skills from the same source and nearby in the registry, with their headline benchmark lift.

Ready to supercharge your AI agents?

See it in action

What's inside

Regression Search

When to use

Usage

Heuristics

Track record

Benchmark

Adoption

SkillSafety

Source

Related skills