---
name: product-experiment-design
source: https://app.decimal.ai/s/product-experiment-design@1/SKILL.md
source_sha256: 1d57725184e5
---

# Design the experiment so the result is decision-grade before it runs

Asked to "set up an A/B test for the new checkout button", the base model writes a fluent, useless
plan: state a hypothesis, split traffic 50/50, track conversion (and time-on-page, and add-to-cart,
and revenue), run it for two weeks, and check for significance. Every one of those choices leaves a
door open for the test to lie. Several metrics with no single decider invites metric-shopping. No
minimum effect means the test can't say whether a "flat" result was a real null or just underpowered.
No guardrails means a conversion win that quietly tanked revenue-per-order ships anyway. And "check
for significance" with no fixed endpoint is an invitation to watch the p-value daily and stop the
moment it dips — which inflates the false-positive rate far past the stated 5%.

A trustworthy experiment is *pre-committed*. The decisions that a motivated analyst could bend after
seeing data — which metric wins, what counts as a win, when to stop — have to be nailed down **before
launch**, when nobody knows which choice favors their preferred outcome. This skill produces that
pre-registration: the design brief that `ab-test-review` later checks the finished test against.

## When to activate

- Someone is about to run an A/B, split, or online experiment and wants it planned so the readout can be trusted.
- "Help me set up / design this test", "how do we run this experiment properly", "make sure this result will be real".
- A change is heading into an experiment and no one has written down the primary metric, the stopping rule, or the guardrails.

Not this skill: judging a test that already ran (ab-test-review), computing the actual number of
users per arm or the exact MDE value from a baseline (power-and-sample-size), agreeing on what a
single metric means to the decimal (metric-definition), or stating a hypothesis with no experiment
attached (falsifiable-hypothesis). This composes with all four — it is the design-side assembly.

## The brief — produce all six

State one only by saying why it does not apply. The value is in *committing*, so pick and write down
a specific choice everywhere a choice exists; do not hand back a menu.

**1. Falsifiable hypothesis with a committed direction.** One sentence naming the change, the
predicted effect, and its direction — plus the result that would prove it wrong. "Moving the buy
button above the fold raises completed purchases; a rate equal or lower than control over the run
falsifies it." A hypothesis that would be "confirmed" by any movement predicts nothing. (For the full
null/alternative/operationalization treatment, that is `falsifiable-hypothesis`.)

**2. Exactly ONE primary metric, with a pre-set decision threshold.** Choose a single metric that
decides ship / no-ship, and say the threshold it must clear (a statistically significant lift of at
least the MDE below). One — not a dashboard. Testing many metrics and shipping on whichever lights up
is multiple-comparisons laundering: at the 0.05 level, one in twenty independent metrics clears by
chance alone. Other metrics may be listed, but explicitly as **secondary / exploratory** — they
inform, they do not decide, and a move on one of them is a hypothesis for next time, not a win.

**3. Minimum detectable effect, committed up front.** State the smallest effect worth acting on — the
lift below which you would not bother shipping — and note that the test must be *powered to detect it*
(typically 80% power at the chosen alpha). This is not the effect you hope for; it is the floor of
"worth it". Its job is twofold: it sets the sample size, and it makes a null result *interpretable* —
"no effect ≥ the MDE" is a real finding, whereas "not significant" from an unspecified-power test says
nothing. Compute the actual n or MDE value with `power-and-sample-size`; here you commit to the number.

**4. Guardrail metrics, each with a regression threshold.** Name the things that must NOT get worse
even if the primary wins — the ways this change could win the battle and lose the war: page latency,
error/crash rate, revenue or margin per user, unsubscribe/opt-out rate, support-ticket volume,
long-run retention. For each, state the threshold that blocks the ship (e.g. "p95 latency must not
rise more than 50 ms"; "revenue-per-session must not drop"). A guardrail with no threshold is decoration.

**5. Stopping rule, fixed before launch.** Decide *now* how the test ends, and never "run until it
hits significance". Two honest forms: (a) a **fixed horizon** — a sample size or duration computed
from the MDE and power, analyzed once at the end; or (b) a **sequential / group-sequential design** —
pre-planned interim looks with an alpha-spending boundary (O'Brien-Fleming, Pocock) or an always-valid
test that budgets the repeated peeks. Name the analysis time(s) and the ship/no-ship rule at each.
Call out the two traps explicitly: peeking daily and stopping on the first p<0.05, and extending a
"nearly significant" test "just a few more days" — both silently inflate the false-positive rate.

**6. Randomization unit and exposure.** State what gets randomized — user, account, session, or a
cluster (e.g. geo/store, when the change spills across users) — and the split (50/50 unless there is a
reason). Match the unit to the metric's grain: a per-user metric randomized by session double-counts.
Analyze only **exposed** units (those who actually hit the change), and expect an assignment/exposure
check (sample-ratio) at analysis time. Note the intended run window and any seasonality/novelty caveat.

## Output

A short brief with the six headings filled in, each committing to a specific choice. If a required
input is missing (baseline rate, the MDE, the traffic), name it as a blocker and state what to
measure — do not invent a plausible number and design around it.