Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →jayrha/ab-test-analyzer
Designs and analyzes A/B tests end-to-end — frames a sharp hypothesis, computes required sample size and test duration, runs significance tests (two-proportion z-test, Welch's t-test, chi-square), reports confidence intervals and lift, and flags common pitfalls like peeking, multiple comparisons, and Simpson's paradox.
Passed 20 of 23 test cases with the skill, and 18 of 23 without it.
2 fixed18 passed either way3 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 2 test cases the skill fixed. Same model, same prompt, with and without the skill.
With 5 million users per arm, our email header test yielded a statistically significant conversion lift from 15.00% to 15.08% (p = 0.002). Discuss whether this result justifies deploying the change.
| Checked | Without skill | With skill |
|---|---|---|
| The response distinguishes statistical significance from practical business impact. | Passed | Passed |
| The response evaluates whether the observed lift meets the pre-specified Minimum Detectable Effect. | Missed | Passed |
The instructions your model reads when the skill loads.
This skill helps design and analyze controlled online experiments (A/B and A/B/n tests) with statistical rigor and honest interpretation. It covers the full lifecycle: hypothesis framing, sample-size and duration planning, choosing the right test, computing p-values and confidence intervals, and avoiding the traps that produce false wins.
Keywords: A/B test, split test, experiment, hypothesis, sample size, power, MDE, minimum detectable effect, statistical significance, p-value, confidence interval, conversion rate, lift, two-proportion z-test, t-test, chi-square, peeking, multiple comparisons, Simpson's paradox, SRM, novelty effect.
Use this skill whenever someone wants to plan an experiment, decide if a result is real, or sanity-check an analysis someone else did.
P(data this extreme or more | null is true) — NOT the probability the null is true, and NOT the probability your variant is better.Follow these steps in order. Do not skip planning steps even when only asked to "analyze results" — verify the plan was sound first.
Frame the hypothesis. Turn the vague goal into a falsifiable statement: "Changing X will increase metric M from baseline b by at least the MDE, because [mechanism]." Identify ONE primary metric (the Overall Evaluation Criterion / OEC). Pre-register guardrail metrics. See references/methodology.md for OEC selection.
Pick the metric type.
Set the design parameters.
b).Compute required sample size and duration. Run scripts/abtest.py size. Divide required N per arm by daily eligible traffic per arm to get duration in days. Round duration UP to full weeks to cover weekday/weekend seasonality (min 1 week, ideally 2).
Run the test untouched for the full pre-planned duration. No peeking-with-action. (You may monitor guardrails for catastrophic harm only.)
Validate data quality BEFORE looking at the result.
scripts/abtest.py srm. If p < 0.001, the experiment is broken — do NOT trust the result.Run the significance test. Use scripts/abtest.py prop (rates) or scripts/abtest.py ttest (means). Report: effect size (absolute + relative lift), confidence interval, p-value, and whether it crossed alpha.
test cases passed, vs 18 / 23 without
Published skills whose descriptions closely match this one, with their headline benchmark lift.
More skills from the same source and nearby in the registry, with their headline benchmark lift.