Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

Ab Test Analysis · phuryn | DecimalAI

phuryn/ab-test-analysis

SkillScore82Maintained

Analyze A/B test results with statistical significance, sample size validation, confidence intervals, and ship/extend/stop recommendations. Use when evaluating experiment results, checking if a test reached significance, interpreting split test data, or deciding whether to ship a variant.

Databy phuryn

Links this skill to your agents and keeps it current with the author’s updates.

Automatic

Data Analyst pack

1 · One-time — install the CLI (docs ↗)

pip install decimalai

2 · Get the skill — writes .claude/skills/phuryn-ab-test-analysis/SKILL.md

decimalai skills pull phuryn-ab-test-analysis --out .claude/skills/

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/phuryn-ab-test-analysis/SKILL.md https://app.decimal.ai/s/phuryn-ab-test-analysis/SKILL.md

Raw file: /s/phuryn-ab-test-analysis/SKILL.md ↗ · machine-readable summary at /s/phuryn-ab-test-analysis.json ↗

In productionGlobal

how this skill behaves on real traffic · last 30 days

No production data for this skill yet — usage numbers appear here once agents start running it.

Anonymized aggregate across every org using this skill — proof from real production use, not a demo.

Does this skill help? · Benchmark: with vs. withoutsign in to run this on your agents

22 test cases from the publisher’s eval.yaml · each run twice on gemini-3.6-flash — with the skill injected, then without · last run 54d ago

Yes — +18% lift · at higher cost+60% tokens0% turns82% of whole cases pass with skill (18/22)

Test case	Without → With	Effect	Δ tokens	Δ turns
case-04	✗→✓	▲ Improved	96%	0%
case-16	✗→✓	▲ Improved	80%	0%
case-17	✗→✓	▲ Improved	51%	0%
case-18	✗→✓	▲ Improved	104%	0%
case-05	✓→✓	＝ Same ✓	47%	0%

Showing 5 of 22 cases

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v4 — 8/3/2026

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

phuryn / pm-skills

pm-data-analytics/skills/ab-test-analysis/SKILL.md

View on GitHub ↗

License MITImported 8/3/2026Version v1

Skill contents

SKILL.md

A/B Test Analysis

Evaluate A/B test results with statistical rigor and translate findings into clear product decisions.

Context

You are analyzing A/B test results for $ARGUMENTS.

If the user provides data files (CSV, Excel, or analytics exports), read and analyze them directly. Generate Python scripts for statistical calculations when needed.

Instructions

Understand the experiment:

What was the hypothesis?
What was changed (the variant)?
What is the primary metric? Any guardrail metrics?
How long did the test run?
What is the traffic split?

Validate the test setup:

Sample size: Is the sample large enough for the expected effect size?
Use the formula: n = (Z²α/2 × 2 × p × (1-p)) / MDE²
Flag if the test is underpowered (<80% power)
Duration: Did the test run for at least 1-2 full business cycles?
Randomization: Any evidence of sample ratio mismatch (SRM)?
Novelty/primacy effects: Was there enough time to wash out initial behavior changes?

Calculate statistical significance:

Conversion rate for control and variant
Relative lift: (variant - control) / control × 100
p-value: Using a two-tailed z-test or chi-squared test
Confidence interval: 95% CI for the difference
Statistical significance: Is p < 0.05?
Practical significance: Is the lift meaningful for the business?

If the user provides raw data, generate and run a Python script to calculate these.

Check guardrail metrics:

Did any guardrail metrics (revenue, engagement, page load time) degrade?
A winning primary metric with degraded guardrails may not be a true win

Interpret results:

| Outcome | Recommendation | |---|---| | Significant positive lift, no guardrail issues | Ship it — roll out to 100% | | Significant positive lift, guardrail concerns | Investigate — understand trade-offs before shipping | | Not significant, positive trend | Extend the test — need more data or larger effect | | Not significant, flat | Stop the test — no meaningful difference detected | | Significant negative lift | Don't ship — revert to control, analyze why |

Provide the analysis summary:

## A/B Test Results: Test Name]

Hypothesis: What we expected] Duration: X days] | Sample: N control / M variant]

| Metric | Control | Variant | Lift | p-value | Significant? | |---|---|---|---|---|---| | Primary] | X% | Y% | +Z% | 0.0X | Yes/No | | Guardrail] | ... | ... | ... | ... | ... |

Recommendation: Ship / Extend / Stop / Investigate] Reasoning: Why] Next steps: What to do]

Think step by step. Save as markdown. Generate Python scripts for calculations if raw data is provided.

Case	Status	Duration (ms)			Turns			Tokens			Tool calls
Case	Status	Without	With	Δ	Without	With	Δ	Without	With	Δ	Without	With	Δ
case-01 I just finished running an experiment on our new checkout CTA color. The control group had 45,000 visitors with 2,250 conversions, while the variant group had 45,200 visitors with 2,480 conversions. Average page latency went from 120ms to 125ms. Can you evaluate these experiment results, run the necessary statistical calculations, and provide a markdown report with a metric summary table, product recommendation, and next steps?	fail→fail	14,086	17,259	+23%	1	1	0%	2,820	3,886	+38%	0	0	—
case-02 We ran a 2-week experiment testing a simplified registration form. Control: 100,000 users, 12.5% signups, 40% 7-day retention. Variant: 100,000 users, 13.8% signups, 38.5% 7-day retention. Our hypothesis was that reducing form fields increases signups. Please review our experiment, check setup health, calculate the relative lift and statistical significance, and return a structured summary with your decision guidance.	fail→fail	21,592	23,355	+8%	1	1	0%	3,942	6,354	+61%	0	0	—
case-03 Here are the metrics from our homepage recommendation module test: Control had 80,000 users with a 5.0% add-to-cart rate. Variant had 80,000 users with a 5.4% add-to-cart rate. Page load time was 1.2s for control vs 1.8s for variant. Please analyze the test validity, compute statistical metrics like p-value and lift, check guardrail impacts, and output a markdown summary containing a metric comparison table and rollout recommendation.	fail→fail	13,963	16,764	+20%	1	1	0%	2,886	3,722	+29%	0	0	—
case-04 We completed a 7-day experiment on our search ranking algorithm. Control had 50,000 queries with a 12.0% click-through rate. Variant had 50,000 queries with a 12.3% click-through rate. The observed p-value is 0.18. Most product team members are eager to roll this out immediately because the trend is positive (+2.5% relative lift). Please analyze these results and provide your rollout recommendation using standard decision guidance.	fail→pass	13,211	26,486	+100%	1	1	0%	2,398	4,703	+96%	0	0	—
case-05 We tested a new subtle drop shadow on our hero banner button for 14 days with 100,000 users per group. Control conversion was 3.10% and variant conversion was 3.11% (p-value = 0.82). Guardrails were completely flat. Some team members want to leave the test running indefinitely until it reaches significance. Evaluate the test and state the recommended rollout outcome.	pass→pass	11,529	15,157	+31%	1	1	0%	2,033	2,992	+47%	0	0	—
case-06 We introduced an aggressive popup modal prompting users to sign up for push notifications during checkout. Control had 20,000 visitors with a 15.0% checkout completion rate. Variant had 20,000 visitors with a 12.2% checkout completion rate (p < 0.001). Provide your evaluation and rollout recommendation based on these findings.	pass→pass	8,740	11,036	+26%	1	1	0%	1,631	3,090	+89%	0	0	—
case-07 We added customer trust badges near the credit card input field on our payment page. Control had 60,000 sessions with 4,200 successful payments (7.0%). Variant had 60,000 sessions with 4,620 successful payments (7.7%). Page load times, refund rates, and customer support tickets remained identical across both arms. Provide a decision report and final rollout guidance.	pass→pass	16,141	12,593	-22%	1	1	0%	2,208	3,648	+65%	0	0	—
case-08 We configured an A/B test on our onboarding flow intended to split traffic evenly (50% control, 50% variant). After 10 days, analytics logs show 65,400 visitors in control and 54,600 visitors in variant. Conversion rates were 8.1% vs 8.9%. Can you assess the validity of this test setup before interpreting the conversion lift?	pass→pass	13,476	15,496	+15%	1	1	0%	2,490	3,936	+58%	0	0	—
case-09 A B2B SaaS startup ran an experiment on their pricing tier display. Control had 150 visitors with 3 conversions (2.0%). Variant had 150 visitors with 5 conversions (3.3%). The team wants to declare victory with a +66% relative lift. Assess whether this experiment setup had sufficient statistical sample power to draw conclusions.	pass→pass	15,859	16,333	+3%	1	1	0%	2,859	4,579	+60%	0	0	—
case-10 An online fashion retailer ran a flash experiment on their homepage layout. The test started on Black Friday morning and ran for 36 hours through Saturday afternoon, collecting 200,000 total visitors and showing a +4% conversion lift (p = 0.02). Evaluate whether this experiment duration is valid for making a permanent product rollout decision.	pass→pass	11,866	37,428	+215%	1	1	0%	2,024	2,869	+42%	0	0	—
case-11 Here is raw daily experiment data for our mobile app nav bar test: Control daily conversions: [120, 115, 130, 125, 118, 122, 128] out of 2000 daily users. Variant daily conversions: [140, 138, 145, 142, 139, 144, 150] out of 2000 daily users. Please calculate the overall conversion rates, lift, and p-value, and output Python code that performs these statistical calculations.	pass→pass	13,915	14,795	+6%	1	1	0%	2,889	4,401	+52%	0	0	—
case-12 An e-commerce mobile app tested showing automated cross-sell popups during add-to-cart. Control (50k users) achieved $4.20 ARPU with a 0.8% app uninstall rate within 24 hours. Variant (50k users) achieved $4.85 ARPU (+15.5% lift, p < 0.001) but the 24-hour uninstall rate surged to 2.4% (p < 0.001). Evaluate these metrics and state the rollout recommendation.	fail→fail	11,295	11,035	-2%	1	1	0%	2,175	3,204	+47%	0	0	—
case-13 We ran a test on our email subscription banner. Control: 50,000 visitors, 2,500 signups (5.0%). Variant: 50,000 visitors, 3,000 signups (6.0%). Calculate the absolute conversion rate difference and its 95% confidence interval.	pass→pass	11,732	12,544	+7%	1	1	0%	2,159	3,756	+74%	0	0	—
case-14 Analyze an A/B test where Control conversion rate is 8.0% (400 / 5,000) and Variant conversion rate is 10.0% (500 / 5,000). Calculate the relative lift percentage explicitly.	pass→pass	11,428	12,278	+7%	1	1	0%	2,730	3,978	+46%	0	0	—
case-15 Evaluate a web button test with Control: 10,000 visitors, 500 conversions (5.0%). Variant: 10,000 visitors, 580 conversions (5.8%). Calculate the two-tailed z-test p-value for the difference in conversion rates.	pass→pass	12,456	15,063	+21%	1	1	0%	2,402	4,365	+82%	0	0	—
case-16 We tested a new free trial banner. Control: 30,000 sessions, 900 starts (3.0%). Variant: 30,000 sessions, 1,080 starts (3.6%). Guardrail latency was unchanged. Provide the summary table adhering to standard experiment reporting format.	fail→pass	11,880	11,041	-7%	1	1	0%	1,931	3,477	+80%	0	0	—
case-17 Provide an A/B test analysis report for our new onboarding survey experiment (Control: 20k users 10% completion, Variant: 20k users 12% completion). Ensure the report starts with the standardized section heading.	fail→pass	10,786	11,956	+11%	1	1	0%	2,292	3,469	+51%	0	0	—
case-18 Evaluate an experiment testing headline copy on our landing page (Control: 15,000 visitors, 3% conversion; Variant: 15,000 visitors, 3.6% conversion; run length 14 days). Provide the experiment summary metadata block.	fail→pass	11,434	16,110	+41%	1	1	0%	1,791	3,655	+104%	0	0	—
case-19 A giant portal tested changing link hover color from dark blue to navy blue. Control had 10,000,000 visitors with a 2.000% click rate. Variant had 10,000,000 visitors with a 2.005% click rate. Due to the huge sample size, the p-value is 0.02 (statistically significant). The relative lift is +0.25%. Evaluate whether this result warrants rolling out the change from a practical product perspective.	pass→pass	16,053	19,966	+24%	1	1	0%	2,155	3,845	+78%	0	0	—
case-20 Write a PostgreSQL query to compute 30-day user retention rates for monthly sign-up cohorts from our `user_events` table.	pass→pass	12,711	14,319	+13%	1	1	0%	2,572	2,956	+15%	0	0	—
case-21 Draft a 5-question user survey to collect qualitative feedback from SaaS subscribers who recently canceled their paid membership.	pass→pass	9,130	9,780	+7%	1	1	0%	1,624	2,487	+53%	0	0	—
case-22 Write a Python script using pandas and statsmodels to fit an ARIMA time-series model on quarterly ARR data to forecast next year's revenue.	pass→pass	11,821	16,932	+43%	1	1	0%	2,349	4,435	+89%	0	0	—

These cases were also screened by the skill’s intent / safety pass.See the Trust review ↗Reproduce this benchmark in Colab ↗

How this skill was measured

DecimalAI ran this skill against gemini-3.6-flash twice over the same eval suite — once with the skill loaded and once without — and compared the two runs case by case. 22 cases were attempted. The headline lift of +18 percentage points is the difference between those two pass rates over the 22 comparable cases.

Model measured: gemini-3.6-flash
Graded by: gemini-3.6-flash
Grading method: judged
Verification: verified
Pass rate without the skill: 63.6%
Pass rate with the skill: 81.8%
Cases attempted: 22
Cases counted toward lift: 22
Cases that regressed: 0
Skill version measured: v1
Run completed: 8/4/2026, 19:17 UTC

Measured example — same prompt, with vs. without the skill · from the benchmark run · case case-04

Prompt

We completed a 7-day experiment on our search ranking algorithm. Control had 50,000 queries with a 12.0% click-through rate. Variant had 50,000 queries with a 12.3% click-through rate. The observed p-value is 0.18. Most product team members are eager to roll this out immediately because the trend is positive (+2.5% relative lift). Please analyze these results and provide your rollout recommendation using standard decision guidance.

Without the skill loaded, the model failed this case. With it loaded, the same prompt on the same model passed. This is one improved case from the latest verified run; every case, including any that regressed, is in the table above.

Show both answers

SkillScore blends 4 signals1 of 4 complete

BenchmarkPass rate WITH skill on a verified run82

Live evalNeeds ≥10 evaluated traces in 30 days

AI ratingNeeds ≥10 rated traces

AdoptionNeeds ≥10 serves across ≥2 orgs in 30 days

Efficiency · separate axis (at equal correctness)+0% turns+60% tokens

Shown for transparency; not credited unless a verified run shows no correctness regression.

Backed by 1 of 4 signals — each added signal makes the score more robust.

Verified on gemini-3.6-flash · re-tested 54 days ago

Other measured skills in the registry, with their headline benchmark lift.