Choose the right product experiment type: superiority, non-inferiority, equivalence, A/B/n, or holdback-backed validation.
SkillScore78Maintained
Automatic
◆Data Analyst pack
Last verified 8/16/2026
What it changes
Passed 17 of 22 test cases with the skill, and 14 of 22 without it.
Without skill65%
With skill78%
Checks it passed that the model alone missed
Recommends a superiority A/B test.
Recommends a non-inferiority test format.
Recommends setting latency as a guardrail metric during backend migration testing.
and 3 more in the full report
4 fixed14 passed either way4 still failing1 broke
Pass-rate lift
+13 pp
Tokens
+4%
Agent turns
0%
Response time
-24%
Model gemini-3.6-flashMeasured 8/16/2026v118 of 23 cases passed with skill · 22 comparable
Tokens, turns and time describe resource use, not a dollar price.
See it in action
One of 3 test cases the skill fixed. Same model, same prompt, with and without the skill.
1 of 3
Prompt
An e-commerce mobile app is migrating its user session backend from Redis to DynamoDB to improve reliability and reduce infrastructure costs. The infrastructure lead suggests running a superiority A/B test to see if purchase conversion increases. What experiment type should actually be recommended?
What was checked, without and with the skill
Checked
Without skill
With skill
Recommends a non-inferiority test.
Passed
Passed
Recommends setting latency as a guardrail metric during backend migration testing.
Missed
Passed
case-11 · gemini-3.6-flash · 8/16/2026 · answers are capped and redacted
What's inside
The instructions your model reads when the skill loads.
SKILL.md79 linessingle file
Experiment Type Selection
Use this skill when the experiment question determines the test type. Not every
experiment should be a simple superiority test; some decisions need evidence
that a change is not worse, roughly equivalent, or durable over time.
Source Traceability
Primary source: Practical A/B Testing by Leemay Nassery. Guidance is
transformed and paraphrased from chapter 3, especially lines 2013-2870. Related
variant design context comes from chapter 1 lines 539-571 and chapter 2 lines
1564-1735.
Related Advanced Skills
experimentation-throughput-strategy: use when the choice is isolated versus
overlapping testing or when testing availability constrains the design.
adaptive-experimentation-strategy: use when fixed-horizon A/B testing may be
replaced by sequential testing, bandits, or contextual bandits.
ml-experiment-evaluation: use when the experiment is evaluating ML models,
rankers, offline metrics, interleaving, or model filtering.
long-term-impact-evaluation: use when the test type question is really
about delayed or sustained impact measurement.
Reference Routing
Need
Read
Test type concepts
references/core/knowledge.md
Selection rules
references/core/rules.md
Scenario examples
references/core/examples.md
Step-by-step selection
workflows/choose-experiment-type.md
Workflow
State the decision question in plain language.
Identify whether the team wants to prove improvement, avoid degradation, or
show practical similarity.
Check whether the metric movement must persist after launch.
Decide whether multiple variants are necessary and interpretable.