Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →hashgraph-online/holdback-experiment-design
Design degradation holdbacks and long-term cumulative holdbacks for product experiments and feature rollouts.
Passed 15 of 23 test cases with the skill, and 3 of 23 without it.
and 6 more in the full report
12 fixed3 passed either way8 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 12 test cases the skill fixed. Same model, same prompt, with and without the skill.
Our team is rolling out an AI writing assistant to all premium SaaS subscribers. Initial 2-week trial data showed high engagement, but we need to know if feature fatigue sets in or if retention improves over a full year. Please draft a full post-rollout holdback strategy document. Make sure to outline the strategic rationale, the holdback mechanism classification, target user cohorts and duration criteria, a metric tracking table detailing roles and review cadences, a breakdown of user, business, and trust risks, and concrete operational rules for when to maintain, terminate, or escalate the holdback.
| Checked | Without skill | With skill |
|---|---|---|
| contains the header '## Holdback Type' | Missed | Passed |
| classifies the holdback type as Degradation | Missed | Passed |
| tracks user retention across a full year in the metrics table | Passed | Passed |
The instructions your model reads when the skill loads.
Use this skill to plan holdbacks that preserve a comparison group after a feature rollout. Holdbacks are useful when effects may appear later, accumulate, or degrade after launch.
Primary source: Practical A/B Testing by Leemay Nassery. Guidance is transformed and paraphrased from chapter 3 lines 2487-2836. Experiment-type context comes from chapter 3 lines 2013-2486.
long-term-impact-evaluation: use before detailed holdback design when the
team should compare holdbacks against post-period analysis, continuous
monitoring, CLV models, or hybrid methods.trustworthy-experiment-insights: use when deciding whether long-term or
holdback evidence is credible enough to drive a product decision.| Need | Read |
|---|---|
| Holdback concepts | references/core/knowledge.md |
| Design and cost rules | references/core/rules.md |
| Scenario examples | references/core/examples.md |
| Step-by-step planning | workflows/design-holdback.md |
test cases passed, vs 3 / 23 without
More skills from the same source and nearby in the registry, with their headline benchmark lift.