Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →hashgraph-online/ml-experiment-evaluation
Plan evaluation strategies for machine-learning product changes.
Passed 23 of 24 test cases with the skill, and 21 of 24 without it.
3 fixed20 passed either way0 still failing1 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 3 test cases the skill fixed. Same model, same prompt, with and without the skill.
We want to use a multi-armed bandit algorithm to dynamically route user traffic to the best news headline model in real time. We update our model feature store once a week via batch jobs and lack real-time feature logging. Should we proceed with the dynamic multi-armed bandit evaluation strategy?
| Checked | Without skill | With skill |
|---|---|---|
| Advises against dynamic multi-armed bandits due to batch feature store updates | Missed | Passed |
| Requires real-time logging infrastructure before deploying adaptive experimentation | Passed | Passed |
The instructions your model reads when the skill loads.
Use this skill to choose how to evaluate machine-learning product changes before they consume live experiment traffic or affect users. It focuses on offline evaluation, offline-online correlation, interleaving, model filtering, and when classic A/B testing or adaptive strategies are justified.
Primary source: Next-Level A/B Testing by Leemay Nassery. Guidance is transformed and paraphrased from Chapter 4 on offline evaluation, offline-online correlation, multi-armed bandits, and interleaving for rankers.
Related skills:
experiment-sensitivity-optimization for reducing live variants and traffic.adaptive-experimentation-strategy for bandits and dynamic allocation.ab-test-design-brief for standard online A/B test planning.| Need | Read |
|---|---|
| ML evaluation concepts | references/core/knowledge.md |
| Selection and validation rules | references/core/rules.md |
| Evaluation strategy examples | references/core/examples.md |
| Step-by-step evaluation plan | workflows/choose-ml-evaluation-strategy.md |
test cases passed, vs 21 / 24 without
More skills from the same source and nearby in the registry, with their headline benchmark lift.