Adaptive Experimentation Strategy

Adaptive Experimentation Strategy

Use this skill to decide whether and how to use adaptive testing strategies instead of fixed-horizon A/B tests. It covers sequential testing, multi-armed bandits, Thompson sampling, contextual bandits, exploration/exploitation, and the data and engineering readiness required to operate them.

Source Traceability

Primary source: Next-Level A/B Testing by Leemay Nassery. Guidance is transformed and paraphrased from Chapter 7 on adaptive testing, sequential testing, multi-armed bandits, Thompson sampling, contextual bandits, and engineering requirements.

Related skills:

• experiment-type-selection for choosing simpler experiment types first.
• ml-experiment-evaluation for model and ranking evaluation paths.
• experiment-verification-monitoring for operational health and alerting.

Reference Routing

Workflow

1. State the decision and why fixed-horizon A/B testing may not be enough.
2. Decide whether the need is early stopping, reward maximization, or personalization.
3. Check data freshness, reward definition, dashboards, on-call ownership, and rollback controls.
4. Choose sequential testing, bandits, Thompson sampling, contextual bandits, or a simpler alternative.
5. Document exploration/exploitation tradeoffs and user/business risk.
6. Define rollout, monitoring, and adoption requirements.