/diagnose — Root-Cause a Wrong or Failing Result

/diagnose — Root-Cause a Wrong or Failing Result

Find why an analysis errors, returns the wrong number, or won't reconcile — with a structured debugging loop rather than scattershot edits. Adapted from the diagnose pattern in mattpocock/skills, reshaped for empirical research code where the bug is usually a silent wrong number, not a crash.

The discipline: never edit before you can reproduce, and never fix before you can explain. A guessed fix that makes the symptom disappear without a named root cause is how a wrong number gets laundered into a published table.

When to use

• A regression / estimate returns a value you can't explain, or one that changed when nothing should have.
• A script errors out and the stack trace doesn't point at the real cause.
• A result "won't reproduce" — different number on re-run, on another machine, or after a package update.
• A replication claim fails /audit-reproducibility and you need to localize which step drifted.

Diagnose is symptom-driven and single-target: ONE wrong number / ONE failing run. Use a sibling instead when the job is different:

• /audit-reproducibility — verify all numeric claims in a manuscript against current code (claim-driven, whole-paper). If you have one FAILing claim and want to localize which pipeline step produced it, /audit-reproducibility hands off to /diagnose; if you want to re-check every table number, start there.
• /review-r — code-quality review with no specific symptom.
• /capture-environment — snapshot the environment when version/seed drift is the suspect.

Phases

Phase 0 — Pin the symptom (expected vs. actual)

State the bug as a falsifiable gap before touching anything:

• Expected: the value/behaviour you believe is correct, and why (a prior run, a paper table, a hand calculation, a theoretical sign).
• Actual: the value/error observed now, copied verbatim (full message, not a paraphrase).
• Tolerance: the threshold that separates "same" from "different", keyed to the source of expected — prior run on the same machine → machine-epsilon + display rounding; a published table → rounding + small slack (~1e-3); a hand calculation → ~0.01; a theoretical prediction → an economic-significance band, not a decimal. Don't chase 1e-12 floating-point noise; don't wave away a 5% gap. (See replication-protocol.md.)

If expected/actual can't be stated, the task is understanding, not diagnosis — stop and clarify first.

Phase 1 — Reproduce deterministically (get a reliable red)

A bug you can't reproduce on demand can't be fixed, only hidden.

1. Fix every source of nondeterminism: set the seed, pin the working directory, record sessionInfo() / pip freeze / Stata version (lean on /capture-environment).
2. Re-run the smallest unit that exhibits the bug and confirm it fails every time. An intermittent failure is its own hypothesis (uninitialised RNG, order-dependent merge, race in parallel code) — note it and carry it into Phase 3.

Phase 2 — Minimise to an MWE