A/B Test Setup

A/B Test Setup

Overview

An A/B test answers one question: did this change move this metric, or was it noise? The answer is only worth something when the sample size, the metric and the stopping rule were fixed before the first visitor arrived. This skill produces three things: a written experiment plan committed before launch, an assignment and logging spec a developer can implement, and a readout that states the effect as an interval instead of a "winner" badge.

Most failed tests fail on arithmetic that could have been done on day zero: the site does not have enough traffic for the effect the team hopes for. Do that arithmetic first.

Instructions

1. Gather the inputs

Ask for what is missing, and look in the project for the rest (feature-flag SDK, analytics events, an experiments/ folder with earlier plans):

• The change and the evidence behind it. Session recordings, support tickets, a funnel drop, a survey. "We want to try it" is allowed, but record that the prior is weak.
• Unit of assignment. Logged-in user, account, or anonymous visitor ID. Accounts when users inside one company would otherwise see different prices or features.
• Primary metric and its baseline, measured over the last 4 full weeks on the same population that will enter the test (same pages, same devices, same traffic sources).
• Eligible units per week — people who actually reach the changed element, not total site traffic.
• Smallest lift worth shipping. This is a business question: below what improvement would nobody bother to keep the variant?
• What else is running on the same pages, and any sale, launch or campaign planned during the run.

2. Do the arithmetic before anything else

For a conversion rate, with control rate p1, variant rate p2 = p1 × (1 + lift) and p̄ their mean, the visitors needed in each arm are:

n = ( z_a × sqrt(2 × p̄ × (1 − p̄)) + z_b × sqrt(p1 × (1 − p1) + p2 × (1 − p2)) )² ÷ (p2 − p1)²

z_a = 1.960   two-sided significance level 0.05
z_b = 0.8416  power 0.80 (an effect of this size is detected 4 times out of 5)

Save the calculator as abtest.py. It also covers the two checks used later.

"""Sample size, sample-ratio check and readout for a two-arm conversion test. Standard library only."""
import math, sys
from statistics import NormalDist

Z = NormalDist()