Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →hashgraph-online/long-term-impact-evaluation
Choose methods for measuring long-term product impact after or beyond an A/B test.
Passed 17 of 23 test cases with the skill, and 14 of 23 without it.
and 5 more in the full report
5 fixed12 passed either way4 still failing2 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 5 test cases the skill fixed. Same model, same prompt, with and without the skill.
We recently launched a new onboarding flow for our fitness app in a 14-day A/B test, which increased initial day-7 retention. However, we're worried that the new flow pushes users into aggressive workout plans that cause burnout and churn around month 3 or 6. We need a strategy to assess the multi-month impact of this change. Please create a long-term impact evaluation plan that states our delayed measurement question, evaluates candidate evaluation approaches in a trade-off table, recommends the optimal method, details our short-term and long-term metrics alongside confounders, and outlines decision rules for when to continue, modify, or escalate the monitoring.
| Checked | Without skill | With skill |
|---|---|---|
| Uses the exact top-level header # Long-Term Impact Evaluation Plan | Missed | Passed |
| Recommends a holdback or continuous monitoring approach while highlighting workout burnout as the delayed effect mechanism | Passed | Passed |
| Includes explicit decision rules covering when to continue measuring, end or revise, and escalate | Passed | Passed |
The instructions your model reads when the skill loads.
Use this skill to choose a practical method for measuring product impact beyond the initial experiment window. It compares long-term holdbacks, post-period analysis, continuous monitoring, and customer lifetime value models.
Primary source: Next-Level A/B Testing by Leemay Nassery. Guidance is transformed and paraphrased from Chapter 8 on long-term impact, short-term and long-term metric relationships, holdbacks, post-period analysis, continuous monitoring, and CLV models.
Related skills:
holdback-experiment-design for detailed holdback planning.trustworthy-experiment-insights for result credibility and false positives.experimentation-strategy-roadmap for cost, quality, and complexity tradeoffs.| Need | Read |
|---|---|
| Long-term evaluation concepts | references/core/knowledge.md |
| Method selection and risk rules | references/core/rules.md |
| Scenario examples | references/core/examples.md |
| Step-by-step method selection | workflows/choose-long-term-evaluation.md |
test cases passed, vs 14 / 23 without
More skills from the same source and nearby in the registry, with their headline benchmark lift.