Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →mkurman/ab-test-setup
When the user wants to plan, design, or implement an A/B test or experiment, or build a growth experimentation program. Also use when the user mentions "A/B test," "split test," "experiment," "test this change," "variant copy," "multivariate test," "hypothesis," "should I test this," "which version is better," "test two versions," "statistical significance," "how long should I run this test," "growth experiments," "experiment velocity," "experiment backlog," "ICE score," "experimentation program
Passed 19 of 22 test cases with the skill, and 10 of 22 without it.
and 11 more in the full report
9 fixed10 passed either way3 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 9 test cases the skill fixed. Same model, same prompt, with and without the skill.
We have an e-commerce product page with a 5% baseline conversion rate. We want to detect a 20% relative lift with 80% power and 95% significance. A team member suggested running with 1,000 visitors per variant. What sample size should we actually target per variant based on quick reference standards?
| Checked | Without skill | With skill |
|---|---|---|
| Specifies 7,000 visitors per variant. | Missed | Passed |
| Rejects 1,000 visitors per variant as insufficient. | Passed | Passed |
The instructions your model reads when the skill loads.
Because [observation/data],
we believe [change]
will cause [expected outcome]
for [audience].
We'll know this is true when [metrics].
Weak: "Changing the button color might increase clicks."
Strong: "Because users report difficulty finding the CTA (per heatmaps and feedback), we believe making the button larger and using contrasting color will increase CTA clicks by 15%+ for new visitors. We'll measure click-through rate from page view to signup start."
| Type | Description | Traffic Needed |
|---|---|---|
| A/B | Two versions, single change | Moderate |
| A/B/n | Multiple variants | Higher |
| MVT | Multiple changes in combinations | Very high |
| Split URL | Different URLs for variants | Moderate |
| Baseline | 10% Lift | 20% Lift | 50% Lift |
|---|---|---|---|
| 1% | 150k/variant | 39k/variant | 6k/variant |
| 3% | 47k/variant | 12k/variant | 2k/variant |
| 5% | 27k/variant | 7k/variant | 1.2k/variant |
test cases passed, vs 10 / 22 without
Published skills whose descriptions closely match this one, with their headline benchmark lift.
More skills from the same source and nearby in the registry, with their headline benchmark lift.