Design the experiment so the result is decision-grade before it runs

Asked to "set up an A/B test for the new checkout button", the base model writes a fluent, useless plan: state a hypothesis, split traffic 50/50, track conversion (and time-on-page, and add-to-cart, and revenue), run it for two weeks, and check for significance. Every one of those choices leaves a door open for the test to lie. Several metrics with no single decider invites metric-shopping. No minimum effect means the test can't say whether a "flat" result was a real null or just underpowered. No guardrails means a conversion win that quietly tanked revenue-per-order ships anyway. And "check for significance" with no fixed endpoint is an invitation to watch the p-value daily and stop the moment it dips — which inflates the false-positive rate far past the stated 5%.

A trustworthy experiment is pre-committed. The decisions that a motivated analyst could bend after seeing data — which metric wins, what counts as a win, when to stop — have to be nailed down before launch, when nobody knows which choice favors their preferred outcome. This skill produces that pre-registration: the design brief that ab-test-review later checks the finished test against.

When to activate

• Someone is about to run an A/B, split, or online experiment and wants it planned so the readout can be trusted.
• "Help me set up / design this test", "how do we run this experiment properly", "make sure this result will be real".
• A change is heading into an experiment and no one has written down the primary metric, the stopping rule, or the guardrails.

Not this skill: judging a test that already ran (ab-test-review), computing the actual number of users per arm or the exact MDE value from a baseline (power-and-sample-size), agreeing on what a single metric means to the decimal (metric-definition), or stating a hypothesis with no experiment attached (falsifiable-hypothesis). This composes with all four — it is the design-side assembly.

The brief — produce all six

State one only by saying why it does not apply. The value is in committing, so pick and write down a specific choice everywhere a choice exists; do not hand back a menu.

1. Falsifiable hypothesis with a committed direction. One sentence naming the change, the predicted effect, and its direction — plus the result that would prove it wrong. "Moving the buy button above the fold raises completed purchases; a rate equal or lower than control over the run falsifies it." A hypothesis that would be "confirmed" by any movement predicts nothing. (For the full null/alternative/operationalization treatment, that is falsifiable-hypothesis.)

2. Exactly ONE primary metric, with a pre-set decision threshold. Choose a single metric that decides ship / no-ship, and say the threshold it must clear (a statistically significant lift of at least the MDE below). One — not a dashboard. Testing many metrics and shipping on whichever lights up is multiple-comparisons laundering: at the 0.05 level, one in twenty independent metrics clears by chance alone. Other metrics may be listed, but explicitly as secondary / exploratory — they inform, they do not decide, and a move on one of them is a hypothesis for next time, not a win.

3. Minimum detectable effect, committed up front. State the smallest effect worth acting on — the lift below which you would not bother shipping — and note that the test must be powered to detect it (typically 80% power at the chosen alpha). This is not the effect you hope for; it is the floor of "worth it". Its job is twofold: it sets the sample size, and it makes a null result interpretable — "no effect ≥ the MDE" is a real finding, whereas "not significant" from an unspecified-power test says nothing. Compute the actual n or MDE value with power-and-sample-size; here you commit to the number.

4. Guardrail metrics, each with a regression threshold. Name the things that must NOT get worse even if the primary wins — the ways this change could win the battle and lose the war: page latency, error/crash rate, revenue or margin per user, unsubscribe/opt-out rate, support-ticket volume, long-run retention. For each, state the threshold that blocks the ship (e.g. "p95 latency must not rise more than 50 ms"; "revenue-per-session must not drop"). A guardrail with no threshold is decoration.

5. Stopping rule, fixed before launch. Decide now how the test ends, and never "run until it hits significance". Two honest forms: (a) a fixed horizon — a sample size or duration computed from the MDE and power, analyzed once at the end; or (b) a sequential / group-sequential design — pre-planned interim looks with an alpha-spending boundary (O'Brien-Fleming, Pocock) or an always-valid test that budgets the repeated peeks. Name the analysis time(s) and the ship/no-ship rule at each. Call out the two traps explicitly: peeking daily and stopping on the first p<0.05, and extending a "nearly significant" test "just a few more days" — both silently inflate the false-positive rate.

6. Randomization unit and exposure. State what gets randomized — user, account, session, or a cluster (e.g. geo/store, when the change spills across users) — and the split (50/50 unless there is a reason). Match the unit to the metric's grain: a per-user metric randomized by session double-counts. Analyze only exposed units (those who actually hit the change), and expect an assignment/exposure check (sample-ratio) at analysis time. Note the intended run window and any seasonality/novelty caveat.

Output

A short brief with the six headings filled in, each committing to a specific choice. If a required input is missing (baseline rate, the MDE, the traffic), name it as a blocker and state what to measure — do not invent a plausible number and design around it.