Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

Get Started Free →

Flaky Test Detector · arabelatso | DecimalAI

testingby arabelatso

Flaky Test Detector

arabelatso/flaky-test-detector

Source arabelatso/skills-4-se ↗License Apache-2.0

Identifies non-deterministic or unreliable tests through static code analysis and test result analysis.

SkillScore92Maintained

Automatic

QA & Test Engineering pack

Last verified 9/19/2026

What it changes

Passed 22 of 24 test cases with the skill, and 16 of 24 without it.

Without skill67%

With skill92%

Checks it passed that the model alone missed

specifies duration as a key in each JSON object
specifies test_name as a key in each JSON object
describes why the pattern causes non-determinism

and 7 more in the full report

7 fixed15 passed either way1 still failing1 broke

Pass-rate lift: +25 pp
Tokens: +82%
Agent turns: 0%
Response time: -2%

Model gemini-3.6-flashMeasured 9/19/2026v122 of 24 cases passed with skill

Tokens, turns and time describe resource use, not a dollar price.

Use this skill

Save it into your project's skills folder. Claude Code picks it up in its next session.

curl -fsSL --create-dirs -o .claude/skills/arabelatso-flaky-test-detector/SKILL.md https://app.decimal.ai/s/arabelatso-flaky-test-detector/SKILL.md

Want it pinned and updatable? pip install "decimalai>=0.15", then decimalai skills add arabelatso-flaky-test-detector, and start Claude Code with claude --plugin-dir "$(git rev-parse --show-toplevel)/.agents"

1 · One-time — install the CLI (docs ↗)

pip install "decimalai>=0.15"

2 · Get the skill — writes .agents/skills/arabelatso-flaky-test-detector/ and pins it in .decimal/skills.lock

decimalai skills add arabelatso-flaky-test-detector

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/arabelatso-flaky-test-detector/SKILL.md https://app.decimal.ai/s/arabelatso-flaky-test-detector/SKILL.md

Raw file: /s/arabelatso-flaky-test-detector/SKILL.md ↗ · machine-readable summary at /s/arabelatso-flaky-test-detector.json ↗

Safety checks: 3 of 3 passed

Code scan CleanIntent PassedContent Clean

See it in action

One of 7 test cases the skill fixed. Same model, same prompt, with and without the skill.

1 of 7

Prompt

We have collected automated CI execution history across 50 runs in a file named `ci_runs.json`. We need to calculate flakiness metrics using the analysis script included in our repository. What is the exact command-line invocation to analyze this JSON file?

What was checked, without and with the skill
Checked	Without skill	With skill
specifies running python scripts/analyze_test_results.py ci_runs.json	Missed	Passed

case-07 · gemini-3.6-flash · 9/19/2026 · answers are capped and redacted

What's inside

The instructions your model reads when the skill loads.

SKILL.md264 lines4 filesincludes scripts

Flaky Test Detector

Identify and fix non-deterministic tests that intermittently fail without code changes.

Quick Start

When a user reports flaky tests or asks for test reliability analysis:

Identify the approach: Determine if analyzing code patterns or test execution results
Analyze for flakiness: Look for common flaky patterns in test code or execution history
Report findings: List identified flaky tests with specific issues
Suggest fixes: Provide concrete remediation strategies

What Makes Tests Flaky

Flaky tests fail intermittently without code changes due to:

Timing issues: Race conditions, fixed sleeps, async/await problems
State management: Shared state between tests, improper cleanup
External dependencies: Network calls, database connections, file system
Randomness: Unseeded random data, UUID generation
Time dependencies: Current time/date, timezone assumptions
Resource issues: Leaks, insufficient cleanup
Test order: Dependencies between tests
Environment: Hardcoded paths, missing env vars

Detection Methods

Static Code Analysis

Analyze test code for common flaky patterns.

When to use:

Reviewing test code for potential issues
Proactive flakiness prevention
Code review of new tests
Refactoring existing tests

Process:

Read test files

Track record

Benchmark

22 / 24

test cases passed, vs 16 / 24 without

Adoption

Users: 5
Times used: 257

Safety and source details

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v5 — 8/27/2026

View 15 scanner checks

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

arabelatso / skills-4-se

skills/flaky-test-detector/SKILL.md

View on GitHub ↗

License Apache-2.0Imported 8/27/2026Version v1Bundled 3 files · scripts

More skills from the same source and nearby in the registry, with their headline benchmark lift.

Ready to supercharge your AI agents?

See it in action

What's inside

Flaky Test Detector

Quick Start

What Makes Tests Flaky

Detection Methods

Static Code Analysis

Track record

Benchmark

Adoption

SkillSafety

Source

Related skills