Flaky Test Triage

A flaky test passes sometimes and fails sometimes on the same code. Shown one, the base model reaches for whatever makes the red go away fastest — wrap it in a retry-until-green, add a longer sleep, or bump the timeout. All three hide the signal without removing the cause: the flake comes back later, and meanwhile a real intermittent bug can be sitting underneath it. This skill does two things instead — name the cause against a fixed taxonomy, then apply a decision rule for whether to quarantine, fix, or delete.

Step 1 — Classify the cause

Every flaky test fails for one of these reasons. Use the signature (what the failure pattern tells you) to pick one; the fix direction follows from the cause.

1. Order / shared state (the victim). Passes when run alone, fails when run after certain other

tests — or only in one shard/execution order. It reads state an earlier test left behind (DB rows, a global, a singleton, a temp file, a cache). Fix direction: make the test set up and tear down its own state so it does not depend on run order.

1. Async / timing. Asserts before an asynchronous operation has settled, or waits a fixed

duration for it. Fails more often on a loaded or slow CI machine and under parallelism; often has a sleep/fixed timeout near the failing assertion. Fix direction: wait for the expected condition to become true (poll/await the state), not for a fixed amount of time.

1. Resource / leak. Fails increasingly as the suite grows or under parallelism, with errors like

"too many open files", "connection pool exhausted", "address already in use", or out-of-memory. An earlier test left connections, file handles, ports, threads, or memory unreleased. Fix direction: release the resource in teardown and bound the pool; the leak is usually not in the failing test.

1. External dependency. Fails when a real network call, third-party API, DNS lookup, or shared

service is slow, rate-limited, or down — the failure correlates with that dependency's health, not with your code. Fix direction: stub or fake the boundary so the test is hermetic; do not test across the network in a unit test.

1. Non-deterministic time or data. Fails at specific times (midnight, month/DST boundaries,

timezones, "today"), or roughly 1-in-N runs with no infra correlation — driven by wall-clock, random/UUIDs/unseeded generators, hash-map or set iteration order, or float rounding. Fix direction: inject a fixed clock and seed; assert order-independently; compare floats with a tolerance.

1. Test pollution (the culprit). This test itself mutates shared state — an env var, a singleton,

a monkeypatch, a global config — and does not restore it, so other tests fail afterward. The inverse of #1: here the flaky symptom shows up in a different test than the one at fault. Fix direction: restore every global it touched in teardown; scope fixtures so mutation cannot escape the test.

Disambiguating quickly: passes alone but fails in the suite → order/shared-state or pollution (which test is at fault — the one that fails, or one that ran before it?). Worse under load or with a sleep present → async/timing. Grows worse as the suite gets bigger, with exhaustion errors → resource/leak. Tracks a third party's uptime → external dependency. Correlates with the calendar or fires ~1/N with no infra pattern → non-deterministic time/data.

Step 2 — Quarantine, fix, or delete

Pick exactly one action. The default when the flake is blocking CI right now is to quarantine first so you stop punishing everyone else's builds — then resolve it properly.

• Quarantine (stopgap, not a resolution). Skip or mark-as-known-flaky and file a tracking ticket

that records the observed signature. This unblocks the pipeline immediately. Quarantine is temporary: a test left quarantined and forgotten is worse than deleted, because it costs maintenance and gives no coverage. It buys time to do the fix; it is not the fix.

• Fix (the real resolution). When the test still covers behavior you need, remove the source of

nondeterminism per the cause you classified (inject the clock/seed, wait on the condition, isolate the state, release the resource, stub the boundary), then un-quarantine it. Every cause in the taxonomy above is fixable — flakiness is a property of the test, not of the world.

• Delete (only with a value justification). Remove the test when it earns removal on its own

merits: it asserts on behavior that no longer exists, it is redundant with another reliable test, or it pins a nondeterministic output/implementation detail that provides no real coverage. Deleting a test because it is flaky is not a justification — that is just hiding the cause. Require a reason the test has no value before deleting.

What never counts as fixing a flake

• A blanket retry-until-green masks the failure and can bury a genuine intermittent bug. Retrying

is a quarantine mechanism at best (unblock + ticket), never a resolution.

• A longer sleep or a bigger timeout moves the timing flake further out; it does not remove the

race. Wait on the condition instead.

• "It passed on rerun, closing it" — a rerun pass is the definition of flaky, not evidence it is

fixed.

Output

State the cause (one of the six), the evidence in the report that points to it, and the action (quarantine / fix / delete) with one line of why. If you quarantine, say it is temporary and that a fix or delete must follow.