---
name: flaky-test-triage
source: https://app.decimal.ai/s/flaky-test-triage@1/SKILL.md
source_sha256: c0b0728e9acf
---

# Flaky Test Triage

A flaky test passes sometimes and fails sometimes on the *same code*. Shown one, the base model
reaches for whatever makes the red go away fastest — wrap it in a retry-until-green, add a longer
`sleep`, or bump the timeout. All three hide the signal without removing the cause: the flake comes
back later, and meanwhile a real intermittent bug can be sitting underneath it. This skill does two
things instead — **name the cause** against a fixed taxonomy, then **apply a decision rule** for
whether to quarantine, fix, or delete.

## Step 1 — Classify the cause

Every flaky test fails for one of these reasons. Use the signature (what the failure pattern tells
you) to pick one; the fix direction follows from the cause.

1. **Order / shared state (the victim).** Passes when run alone, fails when run after certain other
   tests — or only in one shard/execution order. It reads state an earlier test left behind (DB rows,
   a global, a singleton, a temp file, a cache). *Fix direction:* make the test set up and tear down
   its own state so it does not depend on run order.

2. **Async / timing.** Asserts before an asynchronous operation has settled, or waits a *fixed*
   duration for it. Fails more often on a loaded or slow CI machine and under parallelism; often has a
   `sleep`/fixed timeout near the failing assertion. *Fix direction:* wait for the expected condition
   to become true (poll/await the state), not for a fixed amount of time.

3. **Resource / leak.** Fails *increasingly* as the suite grows or under parallelism, with errors like
   "too many open files", "connection pool exhausted", "address already in use", or out-of-memory. An
   earlier test left connections, file handles, ports, threads, or memory unreleased. *Fix direction:*
   release the resource in teardown and bound the pool; the leak is usually not in the failing test.

4. **External dependency.** Fails when a real network call, third-party API, DNS lookup, or shared
   service is slow, rate-limited, or down — the failure correlates with that dependency's health, not
   with your code. *Fix direction:* stub or fake the boundary so the test is hermetic; do not test
   across the network in a unit test.

5. **Non-deterministic time or data.** Fails at specific times (midnight, month/DST boundaries,
   timezones, "today"), or roughly 1-in-N runs with no infra correlation — driven by wall-clock,
   `random`/UUIDs/unseeded generators, hash-map or set iteration order, or float rounding.
   *Fix direction:* inject a fixed clock and seed; assert order-independently; compare floats with a
   tolerance.

6. **Test pollution (the culprit).** This test itself mutates shared state — an env var, a singleton,
   a monkeypatch, a global config — and does not restore it, so *other* tests fail afterward. The
   inverse of #1: here the flaky symptom shows up in a **different** test than the one at fault.
   *Fix direction:* restore every global it touched in teardown; scope fixtures so mutation cannot
   escape the test.

**Disambiguating quickly:** passes alone but fails in the suite → order/shared-state or pollution
(which test is at fault — the one that fails, or one that ran before it?). Worse under load or with a
`sleep` present → async/timing. Grows worse as the suite gets bigger, with exhaustion errors →
resource/leak. Tracks a third party's uptime → external dependency. Correlates with the calendar or
fires ~1/N with no infra pattern → non-deterministic time/data.

## Step 2 — Quarantine, fix, or delete

Pick exactly one action. The default when the flake is **blocking CI right now** is to quarantine
first so you stop punishing everyone else's builds — then resolve it properly.

- **Quarantine (stopgap, not a resolution).** Skip or mark-as-known-flaky *and file a tracking ticket*
  that records the observed signature. This unblocks the pipeline immediately. Quarantine is
  temporary: a test left quarantined and forgotten is worse than deleted, because it costs maintenance
  and gives no coverage. It buys time to do the fix; it is not the fix.

- **Fix (the real resolution).** When the test still covers behavior you need, remove the source of
  nondeterminism per the cause you classified (inject the clock/seed, wait on the condition, isolate
  the state, release the resource, stub the boundary), then un-quarantine it. Every cause in the
  taxonomy above is fixable — flakiness is a property of the test, not of the world.

- **Delete (only with a value justification).** Remove the test when it earns removal on its own
  merits: it asserts on behavior that no longer exists, it is redundant with another reliable test, or
  it pins a nondeterministic output/implementation detail that provides no real coverage. Deleting a
  test *because it is flaky* is not a justification — that is just hiding the cause. Require a reason
  the test has no value before deleting.

## What never counts as fixing a flake

- **A blanket retry-until-green** masks the failure and can bury a genuine intermittent bug. Retrying
  is a quarantine mechanism at best (unblock + ticket), never a resolution.
- **A longer `sleep` or a bigger timeout** moves the timing flake further out; it does not remove the
  race. Wait on the condition instead.
- **"It passed on rerun, closing it"** — a rerun pass is the definition of flaky, not evidence it is
  fixed.

## Output

State the **cause** (one of the six), the **evidence** in the report that points to it, and the
**action** (quarantine / fix / delete) with one line of why. If you quarantine, say it is temporary
and that a fix or delete must follow.