A/B Test Setup

A/B Test Setup

1️⃣ Purpose & Scope

Ensure every A/B test is valid, rigorous, and safe before a single line of code is written.

• Prevents "peeking"
• Enforces statistical power
• Blocks invalid hypotheses

2️⃣ Pre-Requisites

You must have:

• A clear user problem
• Access to an analytics source
• Roughly estimated traffic volume

Hypothesis Quality Checklist

A valid hypothesis includes:

• Observation or evidence
• Single, specific change
• Directional expectation
• Defined audience
• Measurable success criteria

3️⃣ Hypothesis Lock (Hard Gate)

Before designing variants or metrics, you MUST:

• Present the final hypothesis
• Specify:
  • Target audience
  • Primary metric