Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →asgard-ai-platform/stat-ab-testing
Design and analyze A/B tests with proper statistical methodology including sample size calculation, randomization, frequentist and Bayesian approaches, and sequential testing.
Passed 20 of 22 test cases with the skill, and 17 of 22 without it.
and 3 more in the full report
3 fixed17 passed either way2 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 3 test cases the skill fixed. Same model, same prompt, with and without the skill.
I need an A/B test evaluation for a search algorithm tweak we ran on our mobile app. The baseline conversion was 5%, and we targeted a 20% relative MDE. We collected data over two weeks: Control had a 5.0% rate while Treatment achieved 6.1% (p=0.03). Please present this using the standard experiment document structure: start with Hypothesis (H0, H1, primary metric, MDE), followed by Sample Size (baseline, N per group, estimated run time), a post-test Results markdown table detailing Control, Treatment, Diff, 95% CI, and p-value, and conclude with a clear Decision on whether to ship or extend.
| Checked | Without skill | With skill |
|---|---|---|
| Reports a required sample size per group of approximately 15,000 for a 5% baseline rate and 20% relative MDE. | Missed | Passed |
| Uses `## Results (post-test)` for the results section header. | Missed | Passed |
| Concludes with a Decision line formatted as `{Decision} — {rationale}`. | Missed | Passed |
The instructions your model reads when the skill loads.
IRON LAW: Calculate Sample Size BEFORE Running the Test
Running a test without knowing the required sample size leads to two
failures: stopping too early (false positives) or running too long (waste).
Required inputs: baseline conversion rate, minimum detectable effect (MDE),
significance level (α), power (1-β). Calculate BEFORE starting.
n per group ≈ (Z_α/2 + Z_β)² × [p₁(1-p₁) + p₂(1-p₂)] / (p₁ - p₂)²
Quick reference (α=0.05, power=0.8):
| Baseline Rate | MDE (relative) | N per Group |
|---|---|---|
| 5% | 10% (→5.5%) | ~58,000 |
| 5% | 20% (→6.0%) | ~15,000 |
| 10% | 10% (→11%) | ~15,000 |
| 10% | 20% (→12%) | ~4,000 |
| Approach | How It Works | Best When |
|---|---|---|
| Frequentist (fixed-horizon) | Set sample size, run to completion, then analyze | Standard practice, well-understood |
| Bayesian | Update beliefs with data, compute probability of improvement | Want probability statements ("90% chance B is better") |
| Sequential testing | Check results at intervals with adjusted thresholds | Need to stop early if clear winner, or limit downside risk |
test cases passed, vs 17 / 22 without
Published skills whose descriptions closely match this one, with their headline benchmark lift.
More skills from the same source and nearby in the registry, with their headline benchmark lift.