▸case-01 Our growth team wants to launch an A/B test on the pricing page. Halfway through the planned 14-day run, the secondary metric (add-to-cart rate) shows a high p-value improvement, so they want to declare it the new primary success metric instead of checkout completion. Should we allow changing the primary metric mid-experiment? | pass→pass | 16,025 | 22,271 | +39% | 1 | 1 | 0% | 2,061 | 2,362 | +15% | 0 | 0 | — |
▸case-02 We are calculating the required sample size for a standard two-sample proportion test on our landing page redesign. We know our desired significance level is alpha = 0.05, target power is 80%, and minimum detectable effect is 5% relative uplift. What baseline statistical parameter is still missing to compute total sample size? | pass→pass | 5,978 | 4,951 | -17% | 1 | 1 | 0% | 1,206 | 1,323 | +10% | 0 | 0 | — |
▸case-03 Management wants daily status reports showing whether the p-value has crossed 0.05 on an ongoing 30-day experiment, with the intention of stopping early as soon as significance is hit. How should we advise them regarding standard fixed-horizon hypothesis testing? | pass→pass | 14,375 | 17,569 | +22% | 1 | 1 | 0% | 2,368 | 3,014 | +27% | 0 | 0 | — |
▸case-04 We are drafting the pre-experiment plan for a search engine ranking algorithm update. The product manager proposed designating 5 primary metrics (click-through rate, search conversion rate, session duration, query latency, and return rate). What issue arises from having multiple primary metrics? | pass→pass | 13,232 | 13,724 | +4% | 1 | 1 | 0% | 1,986 | 2,543 | +28% | 0 | 0 | — |
▸case-05 We are introducing a complex third-party recommendation widget to our checkout page to increase average order value. What category of metric should we track alongside order value to ensure website performance does not degrade? | pass→pass | 11,076 | 8,039 | -27% | 1 | 1 | 0% | 1,671 | 1,584 | -5% | 0 | 0 | — |
▸case-06 An experiment designer decides to reduce the Minimum Detectable Effect target from a 10% relative change down to a 1% relative change while keeping baseline conversion, alpha, and power constant. How does this decision impact the required sample size? | pass→pass | 6,379 | 7,408 | +16% | 1 | 1 | 0% | 1,358 | 1,612 | +19% | 0 | 0 | — |
▸case-07 Our high-traffic news portal reaches the required sample size of 500,000 visitors per variant in just 18 hours on a Tuesday. Should we stop the experiment after 18 hours? | pass→pass | 14,289 | 12,030 | -16% | 1 | 1 | 0% | 2,207 | 2,052 | -7% | 0 | 0 | — |
▸case-08 When changing the primary navigation bar layout for existing daily active users, we expect a temporary spike in engagement driven purely by curiosity or confusion. How should we structure the metric evaluation to isolate true long-term impact? | fail→pass | 15,568 | 15,454 | -1% | 1 | 1 | 0% | 2,661 | 2,647 | -1% | 0 | 0 | — |
▸case-09 An experiment configured for a 50/50 split logs 52,400 visitors in Variant A and 47,600 visitors in Variant B over one week. Which statistical test should be run to verify if this traffic split indicates a Sample Ratio Mismatch? | pass→pass | 8,213 | 7,060 | -14% | 1 | 1 | 0% | 1,726 | 1,882 | +9% | 0 | 0 | — |
▸case-10 We are testing a 4-step onboarding sign-up wizard across multiple sessions. Should we randomize assignment at the pageview level or at the user account level? | pass→pass | 10,272 | 9,815 | -4% | 1 | 1 | 0% | 1,945 | 2,081 | +7% | 0 | 0 | — |
▸case-11 Before rolling out a major experiment platform upgrade, our data engineering team runs a preliminary trial where both Treatment and Control receive the identical existing production experience. What primary condition should be validated from this trial? | pass→pass | 7,574 | 9,574 | +26% | 1 | 1 | 0% | 1,301 | 1,783 | +37% | 0 | 0 | — |
▸case-12 What standard false positive rate threshold (alpha) is conventionally used in two-tailed fixed-horizon web experimentation? | pass→pass | 6,013 | 2,068 | -66% | 1 | 1 | 0% | 1,019 | 717 | -30% | 0 | 0 | — |
▸case-13 What standard statistical power target (1 minus beta) is conventionally required when calculating sample size for digital product experiments? | pass→pass | 5,874 | 3,040 | -48% | 1 | 1 | 0% | 1,071 | 911 | -15% | 0 | 0 | — |
▸case-14 A copywriter believes a new headline is strictly better than the old headline and suggests using a one-tailed hypothesis test to achieve significance faster with fewer users. Why should we enforce a two-tailed test instead? | pass→pass | 10,314 | 10,617 | +3% | 1 | 1 | 0% | 1,871 | 2,297 | +23% | 0 | 0 | — |
▸case-15 We are running a single experiment comparing 4 distinct variant treatments (A, B, C, D) against a single control group with an overall target alpha of 0.05. Using the standard Bonferroni correction, what adjusted significance threshold per comparison should be used? | pass→pass | 4,856 | 4,597 | -5% | 1 | 1 | 0% | 770 | 1,292 | +68% | 0 | 0 | — |
▸case-16 In a retail app experiment, we must randomize by physical store region rather than individual app user due to store inventory constraints. How does cluster-randomization affect the variance of our metric estimator compared to individual randomization? | pass→pass | 13,764 | 15,049 | +9% | 1 | 1 | 0% | 2,658 | 2,863 | +8% | 0 | 0 | — |
▸case-17 We are replacing our image hosting provider with a cheaper alternative. The goal is to prove the new provider does not slow down page response time by more than 50 milliseconds. What type of hypothesis test should be set up? | pass→pass | 10,441 | 11,377 | +9% | 1 | 1 | 0% | 1,975 | 2,025 | +3% | 0 | 0 | — |
▸case-18 If a product team demands the flexibility to check experiment metrics continuously every afternoon and stop as soon as significance is reached, what statistical method must be implemented prior to starting? | pass→pass | 9,638 | 8,505 | -12% | 1 | 1 | 0% | 1,430 | 1,786 | +25% | 0 | 0 | — |
▸case-19 What core elements must be combined to form a complete, testable experimental hypothesis before setting up an experiment? | pass→pass | 10,418 | 6,657 | -36% | 1 | 1 | 0% | 1,795 | 1,539 | -14% | 0 | 0 | — |
▸case-20 Our A/B test on total revenue yielded an overall p-value of 0.35 (inconclusive). The product manager filtered the post-experiment data across 40 different user demographic breakdowns and found that mobile web users in Germany aged 18-24 had a p-value of 0.02. Can we report this subgroup finding as a proven positive result? | pass→pass | 10,263 | 9,766 | -5% | 1 | 1 | 0% | 1,837 | 2,119 | +15% | 0 | 0 | — |
▸case-21 We are running a 24-hour flash sale with 5 promotional hero banners and want to dynamically shift traffic to whichever banner is converting best during the sale to maximize immediate revenue. Should we use a fixed-horizon A/B test or a multi-armed bandit algorithm? | pass→pass | 12,240 | 12,137 | -1% | 1 | 1 | 0% | 2,047 | 2,373 | +16% | 0 | 0 | — |
▸case-22 Our company ran a nationwide billboard advertising campaign where user-level random assignment was impossible. How can we estimate the causal effect of the campaign on website traffic using observational control regions? | pass→pass | 16,568 | 20,581 | +24% | 1 | 1 | 0% | 2,861 | 3,776 | +32% | 0 | 0 | — |