Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

Scenario-Driven Testing for AI Code Generation · aiskillstore | DecimalAI

aiskillstore/scenario-testing

SkillScore95Maintained

This skill should be used when writing tests, validating features, or needing to verify code works. Triggers on "write tests", "add test coverage", "validate feature", "integration test", "end-to-end", "e2e test", "mock", "unit test". Enforces scenario-driven testing with real dependencies in .scratch/ directory.

Testingby aiskillstore

Links this skill to your agents and keeps it current with the author’s updates.

Automatic

QA & Test Engineering pack

1 · One-time — install the CLI (docs ↗)

pip install decimalai

2 · Get the skill — writes .claude/skills/aiskillstore-scenario-testing/SKILL.md

decimalai skills pull aiskillstore-scenario-testing --out .claude/skills/

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/aiskillstore-scenario-testing/SKILL.md https://app.decimal.ai/s/aiskillstore-scenario-testing/SKILL.md

Raw file: /s/aiskillstore-scenario-testing/SKILL.md ↗ · machine-readable summary at /s/aiskillstore-scenario-testing.json ↗

In productionGlobal

how this skill behaves on real traffic · last 30 days

No production data for this skill yet — usage numbers appear here once agents start running it.

Anonymized aggregate across every org using this skill — proof from real production use, not a demo.

Does this skill help? · Benchmark: with vs. withoutsign in to run this on your agents

22 test cases from the publisher’s eval.yaml · each run twice on gemini-3.6-flash — with the skill injected, then without · last run 31d ago

Yes — +59% lift-13% tokens0% turns95% of whole cases pass with skill (21/22)

Test case	Without → With	Effect	Δ tokens	Δ turns
case-01	✗→✓	▲ Improved	39%	0%
case-02	✗→✓	▲ Improved	-31%	0%
case-03	✗→✓	▲ Improved	8%	0%
case-04	✗→✓	▲ Improved	-11%	0%
case-05	✗→✓	▲ Improved	-31%	0%

Showing 5 of 22 cases

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v4 — 8/3/2026

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

aiskillstore / marketplace

skills/2389-research/scenario-testing/SKILL.md

View on GitHub ↗

Imported 8/3/2026Version v1

Skill contents

SKILL.md

Scenario-Driven Testing for AI Code Generation

Core Principle

The Iron Law: "NO FEATURE IS VALIDATED UNTIL A SCENARIO PASSES WITH REAL DEPENDENCIES"

Mocks create false confidence. Only scenarios exercising real systems validate that code works.

The Truth Hierarchy

Scenario tests (real system, real data) = truth
Unit tests (isolated) = human comfort only
Mocks = lies hiding bugs

As stated in the principle: "A test that uses mocks is not testing your system. It's testing your assumptions about how dependencies behave."

When to Use This Skill

Validating new functionality
Before declaring work complete
When tempted to use mocks
After fixing bugs requiring verification
Any time you need to prove code works

Required Practices

1. Write Scenarios in `.scratch/`

Use any language appropriate to the task
Exercise the real system end-to-end
Zero mocks allowed
Must be in .gitignore (never commit)

2. Promote Patterns to `scenarios.jsonl`

Extract recurring scenarios as documented specifications
One JSON line per scenario
Include: name, description, given/when/then, validates
This file IS committed

3. Use Real Dependencies

External APIs must hit actual services (sandbox/test mode acceptable). Mocking any dependency invalidates the scenario.

4. Independence Requirement

Each scenario must run standalone without depending on prior executions. This enables:

Parallel execution
Prevents hidden ordering dependencies
Reliable CI/CD integration

What Makes a Scenario Invalid

A scenario is invalid if it:

Contains any mocks whatsoever
Uses fake data instead of real storage
Depends on another scenario running first
Never actually executed to verify it passes

Common Violations to Avoid

Reject these rationalizations:

"Just a quick unit test..." - Unit tests don't validate features
"Too simple for end-to-end..." - Integration breaks simple things
"I'll mock for speed..." - Speed doesn't matter if tests lie
"I don't have API credentials..." - Ask your human partner for real ones

Definition of Done

A feature is complete only when:

✅ A scenario in .scratch/ passes with zero mocks
✅ Real dependencies are exercised
✅ .scratch/ remains in .gitignore
✅ Robust patterns extracted to scenarios.jsonl

Example Workflow

Write scenario - Create .scratch/test-user-registration.py
Use real dependencies - Hit real database, real auth service (test mode)
Run and verify - Execute scenario, confirm it passes
Extract pattern - Document in scenarios.jsonl
Keep .scratch ignored - Never commit scratch scenarios

Why This Matters

Unit tests verify isolated logic
Integration tests verify components work together
Scenario tests verify the system actually works

Only scenario tests prove your feature delivers value to users.

Illustrative Want to see what aiskillstore/scenario-testing would put in front of your model? Preview the rendered body and check its trigger phrases — no LLM call, no signup.

Latest benchmark · publisher runVerified

8/21/2026, 19:03 UTC · 22 cases

Lift

+59%

95% vs 36%

Agent turns

1.0 vs 1.0

Tokens

-13%

1,637.05 vs 1,887

fail → pass · 13pass → pass · 8fail → fail · 1

Per-case results (22)

Case	Status	Duration (ms)			Turns			Tokens			Tool calls
Case	Status	Without	With	Δ	Without	With	Δ	Without	With	Δ	Without	With	Δ
case-22 When writing standard unit tests for pure internal helper functions that format dates without any external I/O, does the zero-mock scenario testing requirement apply to these unit tests?	fail→fail	13,265	10,204	-23%	1	1	0%	2,058	2,218	+8%	0	0	—
case-01 I just finished building the Stripe payment webhook handler. I need to verify that this feature works properly before marking it complete. Please write a standalone test script to run this flow against actual test-mode endpoints, and provide a structured JSON specification line that I can check into git.	fail→pass	11,056	10,753	-3%	1	1	0%	2,018	2,805	+39%	0	0	—
case-02 I fixed a critical bug in our user signup pipeline where database constraints were failing. Create a validation script that exercises the full signup process with a real local database, and format a reusable scenario entry detailing the given/when/then conditions for our test registry.	fail→pass	33,101	9,987	-70%	1	1	0%	3,734	2,575	-31%	0	0	—
case-03 We are adding an external email alert integration and need to prove it works end-to-end. Can you generate the initial test script using our test API keys, and explain how to structure both the temporary execution file and the committed scenario tracking record?	fail→pass	14,812	12,116	-18%	1	1	0%	2,706	2,917	+8%	0	0	—
case-04 We are formalizing our scenario registry file for automated parsing. Should each scenario entry in scenarios.jsonl be a formatted multi-line JSON object with pretty indentation, or structured as one JSON object per line?	fail→pass	9,391	3,919	-58%	1	1	0%	1,531	1,360	-11%	0	0	—
case-05 We are setting up our scenario test directory layout. Where should temporary executable test scripts be stored during active development, and how should git handle that directory?	fail→pass	9,672	3,097	-68%	1	1	0%	1,690	1,171	-31%	0	0	—
case-06 When promoting a validated test scenario into scenarios.jsonl, what specific schema fields must be present in the JSON record to document the test conditions and validation target?	fail→pass	11,760	4,036	-66%	1	1	0%	2,066	1,292	-37%	0	0	—
case-07 A developer on our team wants to mock the S3 storage bucket in our file upload validation script to make the test run faster in CI. Should we accept this pull request mock approach for validating the upload feature?	fail→pass	12,231	5,295	-57%	1	1	0%	1,863	1,516	-19%	0	0	—
case-08 We have five scenario tests for order processing. Should scenario B assume that scenario A has already populated the database with a test customer, or should each scenario execute independently?	pass→pass	10,154	3,472	-66%	1	1	0%	1,512	1,226	-19%	0	0	—
case-09 What conditions must be satisfied before a newly implemented software feature can be declared complete and ready for production release under scenario-based validation standards?	fail→pass	13,480	5,948	-56%	1	1	0%	2,086	1,613	-23%	0	0	—
case-10 When evaluating test confidence across scenario tests, isolated unit tests, and test doubles/mocks, how do these test types rank in terms of providing true validation that a feature works?	pass→pass	14,298	5,142	-64%	1	1	0%	2,139	1,518	-29%	0	0	—
case-11 A developer cannot run the scenario test for payment gateway integration because they lack API credentials for the test environment and are considering writing a mock client instead. What should the developer do?	fail→pass	9,315	5,385	-42%	1	1	0%	1,408	1,268	-10%	0	0	—
case-12 When converting a temporary test script from .scratch/ into a permanent scenario record in scenarios.jsonl, should the script in .scratch/ be committed to git?	pass→pass	7,685	2,869	-63%	1	1	0%	1,274	1,167	-8%	0	0	—
case-13 Why is ordering independence between scenario tests required when building CI/CD test pipelines?	pass→pass	12,122	7,740	-36%	1	1	0%	1,965	1,730	-12%	0	0	—
case-14 Are developers restricted to using Python when writing temporary end-to-end scenario scripts in .scratch/, or can they select other programming languages?	pass→pass	10,498	2,527	-76%	1	1	0%	1,617	1,067	-34%	0	0	—
case-15 When running scenario tests against third-party SaaS APIs like Twilio or SendGrid, is it permissible to hit sandbox or test-mode endpoints, or must live production accounts be used?	pass→pass	12,053	5,188	-57%	1	1	0%	1,806	1,496	-17%	0	0	—
case-16 A candidate scenario test uses an in-memory dictionary array to simulate database state instead of reading from a real database instance. Is this scenario test valid?	fail→pass	11,597	5,173	-55%	1	1	0%	1,773	1,286	-27%	0	0	—
case-17 A developer wrote a scenario test script and documented it in scenarios.jsonl, but never actually ran the execution script to verify that it succeeds. Is this scenario considered valid?	pass→pass	9,705	3,084	-68%	1	1	0%	1,416	1,180	-17%	0	0	—
case-18 After resolving a critical regression in a service, what type of test verification must be performed before declaring the bug fix verified?	fail→pass	15,250	4,524	-70%	1	1	0%	1,327	1,384	+4%	0	0	—
case-19 A team member argues that a simple utility feature is too small for end-to-end scenario testing and that a quick unit test with mocked dependencies is sufficient. How should the team respond?	fail→pass	12,809	7,092	-45%	1	1	0%	1,902	1,794	-6%	0	0	—
case-20 We are writing standard unit tests for an in-memory utility function that parses JSON strings. Should these standard unit tests be placed in .scratch/ alongside temporary scenario test scripts?	pass→pass	11,944	4,752	-60%	1	1	0%	1,349	1,474	+9%	0	0	—
case-21 We are writing automated load testing scripts using a load testing tool to measure peak request volume and response latency. Should each load test run be appended as a scenario entry to scenarios.jsonl?	fail→pass	13,500	8,343	-38%	1	1	0%	2,274	1,958	-14%	0	0	—

These cases were also screened by the skill’s intent / safety pass.See the Trust review ↗Reproduce this benchmark in Colab ↗

How this skill was measured

DecimalAI ran this skill against gemini-3.6-flash twice over the same eval suite — once with the skill loaded and once without — and compared the two runs case by case. 22 cases were attempted. The headline lift of +59 percentage points is the difference between those two pass rates over the 22 comparable cases.

Model measured: gemini-3.6-flash
Graded by: gemini-3.6-flash
Grading method: judged
Verification: verified
Pass rate without the skill: 36.4%
Pass rate with the skill: 95.5%
Cases attempted: 22
Cases counted toward lift: 22
Cases that regressed: 0
Skill version measured: v1
Run completed: 8/21/2026, 19:03 UTC

Measured example — same prompt, with vs. without the skill · from the benchmark run · case case-01

Prompt

I just finished building the Stripe payment webhook handler. I need to verify that this feature works properly before marking it complete. Please write a standalone test script to run this flow against actual test-mode endpoints, and provide a structured JSON specification line that I can check into git.

Without the skill loaded, the model failed this case. With it loaded, the same prompt on the same model passed. This is one improved case from the latest verified run; every case, including any that regressed, is in the table above.

Show both answers

SkillScore blends 4 signals1 of 4 complete

BenchmarkPass rate WITH skill on a verified run95

Live evalNeeds ≥10 evaluated traces in 30 days

AI ratingNeeds ≥10 rated traces

AdoptionNeeds ≥10 serves across ≥2 orgs in 30 days

Efficiency · separate axis (at equal correctness)+0% turns-17% tokens

Shown for transparency; not credited unless a verified run shows no correctness regression.

Backed by 1 of 4 signals — each added signal makes the score more robust.

Verified on gemini-3.6-flash · re-tested 30 days ago

Other measured skills in the registry, with their headline benchmark lift.