Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →product-on-purpose/measure-experiment-design
Designs an A/B test or experiment with variants, success metrics, sample size, and duration for an existing hypothesis.
Passed 18 of 22 test cases with the skill, and 11 of 22 without it.
and 9 more in the full report
7 fixed11 passed either way4 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 7 test cases the skill fixed. Same model, same prompt, with and without the skill.
We surveyed 500 cancelled subscribers using a 5-point Likert scale and open-ended text fields asking why they left our streaming platform. Here are the raw survey response rows. Can you run a survey analysis to extract sentiment clusters and top cancellation reasons?
| Checked | Without skill | With skill |
|---|---|---|
| States that analyzing qualitative survey responses is outside controlled experiment design scope. | Missed | Passed |
| Refrains from outputting an A/B experiment design spec. | Passed | Passed |
The instructions your model reads when the skill loads.
An experiment design document defines all parameters needed to run a rigorous A/B test or controlled experiment. It ensures the team aligns on what you're testing, how you'll measure success, and how long to run the test before drawing conclusions. Good experiment design prevents common pitfalls: underpowered tests, unclear success criteria, and decisions based on noise rather than signal.
define-hypothesis first; this skill designs the test for a claim you already havemeasure-experiment-resultsmeasure-instrumentation-specmeasure-survey-analysisWhen asked to design an experiment, follow these steps:
Articulate the Hypothesis Write a clear, testable hypothesis in the format: "We believe [change] for [users] will [outcome] as measured by [metric]." One hypothesis per experiment - if you're testing multiple things, run multiple experiments.
Define the Variants Describe the control (current experience) and treatment (new experience) in sufficient detail. Include screenshots, mockups, or precise descriptions so anyone can understand what users will see.
Choose Primary and Secondary Metrics Select one primary metric that will determine success or failure. Add 2-3 secondary metrics to understand the broader impact. Include guardrail metrics to catch unintended negative effects.
Calculate Sample Size Determine how many users you need per variant to detect your minimum detectable effect (MDE) with statistical significance. Specify your significance level (typically 0.05) and power (typically 0.80).
Estimate Duration Based on sample size and available traffic, calculate how long the experiment needs to run. Account for weekly patterns - avoid ending mid-week if behavior varies by day.
Define Targeting and Allocation
test cases passed, vs 11 / 22 without
Published skills whose descriptions closely match this one, with their headline benchmark lift.
More skills from the same source and nearby in the registry, with their headline benchmark lift.