Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →hashgraph-online/experiment-sensitivity-optimization
Improve experiment sensitivity and reduce traffic or duration requirements.
Passed 17 of 22 test cases with the skill, and 12 of 22 without it.
and 3 more in the full report
5 fixed12 passed either way5 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 5 test cases the skill fixed. Same model, same prompt, with and without the skill.
Our A/B test for the new checkout payment flow is running way too slowly because purchase amount variance is extremely high across users. Can you help redesign this test so we can detect real impact faster? Please return a markdown report formatted as an 'Experiment Sensitivity Plan' that includes the core decision, our current constraint, a table listing recommended changes alongside why they help, requirements, and risks, a metric plan (with primary metric, a more sensitive proxy, guardrails, and minimum detectable effect), a variance reduction breakdown (technique, data needed, validation), and interpretation notes covering what the design can and cannot conclude.
| Checked | Without skill | With skill |
|---|---|---|
| The document includes the exact section header `## Current Constraint`. | Missed | Passed |
| The plan recommends CUPED variance reduction to address high purchase amount variance. | Passed | Passed |
The instructions your model reads when the skill loads.
Use this skill to redesign an experiment so it can detect meaningful effects with fewer users, less time, or clearer metrics. It focuses on minimum detectable effect, metric sensitivity, capping, variant reduction, CUPED, and variance reduction.
Primary source: Next-Level A/B Testing by Leemay Nassery. Guidance is transformed and paraphrased from Chapter 3 on experiment design, sensitive metrics, minimum detectable effect, capping, reducing variants, and CUPED; and Chapter 6 on stratified random sampling and covariate adjustments.
Related skills:
ab-test-design-brief for baseline experiment specs.trustworthy-experiment-insights for judging whether a result is believable.experimentation-throughput-strategy for capacity and test scheduling.| Need | Read |
|---|---|
| Sensitivity concepts | references/core/knowledge.md |
| Metric, variance, and sample-size rules | references/core/rules.md |
| Optimization scenarios | references/core/examples.md |
| Step-by-step sensitivity review | workflows/optimize-experiment-sensitivity.md |
test cases passed, vs 12 / 22 without
More skills from the same source and nearby in the registry, with their headline benchmark lift.