Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

AI Regression Testing · loulanyue | DecimalAI

testingby loulanyue

AI Regression Testing

loulanyue/ai-regression-testing

Source loulanyue/awesome-claude-notes ↗License MIT

Regression testing strategies for AI-assisted development. Sandbox-mode API testing without database dependencies, automated bug-check workflows, and patterns to catch AI blind spots where the same model writes and reviews code.

SkillScore92Maintained

Automatic

QA & Test Engineering pack

Last verified 8/21/2026

What it changes

Passed 23 of 25 test cases with the skill, and 14 of 25 without it.

Without skill56%

With skill92%

Checks it passed that the model alone missed

References the parseResponse test helper function.
Defines the createTestRequest test helper function.
References the createTestRequest test helper function.

and 12 more in the full report

9 fixed14 passed either way2 still failing0 broke

Pass-rate lift: +36 pp
Tokens: +105%
Agent turns: 0%
Response time: -33%

Model gemini-3.6-flashMeasured 8/21/2026v723 of 25 cases passed with skill

Tokens, turns and time describe resource use, not a dollar price.

Use this skill

Save it into your project's skills folder. Claude Code picks it up in its next session.

curl -fsSL --create-dirs -o .claude/skills/loulanyue-ai-regression-testing/SKILL.md https://app.decimal.ai/s/loulanyue-ai-regression-testing/SKILL.md

Want it pinned and updatable? pip install "decimalai>=0.15", then decimalai skills add loulanyue-ai-regression-testing, and start Claude Code with claude --plugin-dir "$(git rev-parse --show-toplevel)/.agents"

1 · One-time — install the CLI (docs ↗)

pip install "decimalai>=0.15"

2 · Get the skill — writes .agents/skills/loulanyue-ai-regression-testing/ and pins it in .decimal/skills.lock

decimalai skills add loulanyue-ai-regression-testing

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/loulanyue-ai-regression-testing/SKILL.md https://app.decimal.ai/s/loulanyue-ai-regression-testing/SKILL.md

Raw file: /s/loulanyue-ai-regression-testing/SKILL.md ↗ · machine-readable summary at /s/loulanyue-ai-regression-testing.json ↗

Safety checks: 3 of 3 passed

Code scan CleanIntent PassedContent Clean

See it in action

One of 9 test cases the skill fixed. Same model, same prompt, with and without the skill.

1 of 9

Prompt

We want a helper function `createTestRequest` for creating `NextRequest` instances in Vitest to test Next.js App Router handlers. When testing sandbox user scenarios, how should user impersonation be passed in the request headers?

What was checked, without and with the skill
Checked	Without skill	With skill
Sets the x-sandbox-user-id HTTP header when a sandbox user identifier option is provided.	Missed	Passed

case-08 · gemini-3.6-flash · 8/21/2026 · answers are capped and redacted

What's inside

The instructions your model reads when the skill loads.

SKILL.md386 linessingle file

AI Regression Testing

Testing patterns specifically designed for AI-assisted development, where the same model writes code and reviews it — creating systematic blind spots that only automated tests can catch.

When to Activate

AI agent (Claude Code, Cursor, Codex) has modified API routes or backend logic
A bug was found and fixed — need to prevent re-introduction
Project has a sandbox/mock mode that can be leveraged for DB-free testing
Running /bug-check or similar review commands after code changes
Multiple code paths exist (sandbox vs production, feature flags, etc.)

The Core Problem

When an AI writes code and then reviews its own work, it carries the same assumptions into both steps. This creates a predictable failure pattern:

AI writes fix → AI reviews fix → AI says "looks correct" → Bug still exists

Real-world example (observed in production):

Fix 1: Added notification_settings to API response
  → Forgot to add it to the SELECT query
  → AI reviewed and missed it (same blind spot)

Fix 2: Added it to SELECT query
  → TypeScript build error (column not in generated types)
  → AI reviewed Fix 1 but didn't catch the SELECT issue

Fix 3: Changed to SELECT *
  → Fixed production path, forgot sandbox path
  → AI reviewed and missed it AGAIN (4th occurrence)

Fix 4: Test caught it instantly on first run ✅

The pattern: sandbox/production path inconsistency is the #1 AI-introduced regression.

Track record

Benchmark

23 / 25

test cases passed, vs 14 / 25 without

Adoption

Users: 4
Times used: 59

Safety and source details

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v4 — 8/3/2026

View 15 scanner checks

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

loulanyue / awesome-claude-notes

skills/ai-regression-testing/SKILL.md

View on GitHub ↗

License MITImported 8/3/2026Version v7

Other skills for the same job

Published skills whose descriptions closely match this one, with their headline benchmark lift.

More skills from the same source and nearby in the registry, with their headline benchmark lift.

Ready to supercharge your AI agents?

See it in action

What's inside

AI Regression Testing

When to Activate

The Core Problem

Track record

Benchmark

Adoption

SkillSafety

Source

Other skills for the same job

Related skills