Analyze and communicate A/B test results with metric readouts, subgroup analysis, data-quality checks, ad hoc investigation, visualization, and launch recommendations.
SkillScore95Maintained
Automatic
◆Data Analyst pack
Last verified 8/15/2026
What it changes
Passed 21 of 22 test cases with the skill, and 15 of 22 without it.
Without skill68%
With skill95%
Checks it passed that the model alone missed
The summary section includes a run window field.
Selects Iterate as the executive decision status.
The Segment Findings section is formatted as a table.
and 5 more in the full report
7 fixed14 passed either way0 still failing1 broke
Pass-rate lift
+27 pp
Tokens
+6%
Agent turns
0%
Response time
-25%
Model gemini-3.6-flashMeasured 8/15/2026v121 of 22 cases passed with skill
Tokens, turns and time describe resource use, not a dollar price.
See it in action
One of 7 test cases the skill fixed. Same model, same prompt, with and without the skill.
1 of 7
Prompt
A new search filter interface yielded a non-significant overall conversion change (+0.2%), but power users (top 10% query frequency) showed a planned +4.5% conversion gain. The team wants to refine the UI for passive searchers. Draft the executive decision for this search filter experiment.
What was checked, without and with the skill
Checked
Without skill
With skill
Selects Iterate as the executive decision status.
Missed
Passed
Explains that overall conversion was neutral while power user conversion improved.
Passed
Passed
case-10 · gemini-3.6-flash · 8/15/2026 · answers are capped and redacted
What's inside
The instructions your model reads when the skill loads.
SKILL.md83 linessingle file
A/B Test Results Readout
Use this skill to turn experiment data into a clear decision. It emphasizes
metric interpretation, data-quality checks, subgroup analysis, ad hoc analysis,
visualization, and launch recommendations.
Source Traceability
Primary source: Practical A/B Testing by Leemay Nassery. Guidance is
transformed and paraphrased from chapter 4 lines 2950-3742 and chapter 1 lines
572-718. Metric tradeoff context comes from chapter 2 lines 1296-1472.
Related Advanced Skills
trustworthy-experiment-insights: use when the readout needs false positive,
false negative, power, replication, meta-analysis, or suspicious-lift review.
experiment-verification-monitoring: use when result interpretation depends
on whether assignment, exposure, metrics, canaries, or active monitoring were
healthy.
long-term-impact-evaluation: use when short-term readout is not enough to
decide durable product or business impact.
Reference Routing
Need
Read
Readout concepts
references/core/knowledge.md
Analysis and reporting rules
references/core/rules.md
Example readouts
references/core/examples.md
Step-by-step report workflow
workflows/prepare-results-readout.md
Workflow
Reconstruct the test design: hypothesis, variants, population, and metrics.
Verify data quality and whether exposure/eligibility match the brief.
Compare primary and guardrail metrics against baseline and decision rules.
Run subgroup analysis when averages obscure meaningful differences.
Investigate outliers, missing data, or surprising movement.
Visualize results so stakeholders can compare control, test, and segments.
Recommend ship, stop, iterate, or investigate with caveats.