Managing experiment lifecycle

This skill covers experiment state transitions — what each action does, when to use it, and how it affects variant assignment and analysis.

State diagram

text
draft ──launch──▶ running ──end──▶ stopped ──archive──▶ archived
                    │   ▲              │
                  pause resume    ship_variant
                    │   │         (also ends if running)
                    ▼   │
                  paused (flag inactive, still "running" status)

Any non-draft state ──reset──▶ draft

Actions and their implications

For each action, the two key questions:

1. Who sees what variant? (user perspective)
2. Who is in my analysis? (statistical perspective)

Launch (experiment-launch)

Transitions draft → running. Activates the feature flag and sets start_date.

• Preconditions: must be in draft, flag needs ≥2 variants with "control" first
• Pre-launch checklist: has at least one metric? Variants correct? Flag implemented in code?
• Variants: users start being bucketed into variants based on the configured split
• Analysis: data collection begins from start_date

No request body needed.

Pause (experiment-pause)

Deactivates the feature flag. Users fall back to the default experience (typically control).

• Preconditions: must be running and not already paused
• Variants: flag is not returned by /decide — no new exposure events recorded
• Analysis: no new data while paused, but existing data is preserved. Experiment stays "running".

No request body. Use experiment-resume to reactivate.

Resume (experiment-resume)

Reactivates the feature flag after a pause. Users are re-bucketed deterministically into the same variants.

• Preconditions: must be paused
• Variants: same assignment as before pause — deterministic bucketing
• Analysis: exposure tracking resumes

No request body.

End (experiment-end)

Sets end_date and transitions to stopped. The feature flag is NOT modified.

• Preconditions: must be running (launched, not already stopped)
• Variants: users continue seeing assigned variants (flag stays active)
• Analysis: results frozen to data up to end_date

Optional body: conclusion ("won", "lost", "inconclusive", "stopped_early", "invalid") and conclusion_comment.

Use this when you want to freeze results without changing what users see.

Ship variant (experiment-ship-variant)

Rewrites the feature flag so the selected variant is served to 100% of users.

• Preconditions: must be launched (running or stopped). Cannot ship from draft.
• Variants: ALL users see the shipped variant. The flag is rewritten with a catch-all group.
• Analysis: if still running, the experiment is also ended (end_date set)

Always confirm with the user before shipping — this permanently rewrites the feature flag.

Required: variant_key (e.g. "test"). Optional: conclusion, conclusion_comment.

Returns 409 if an approval policy requires review before the flag change.

Archive (experiment-archive)

Hides a stopped experiment from the default list view.

• Preconditions: must be stopped (end_date set)
• Variants: no change — flag is unaffected
• Analysis: no change — results remain accessible

No request body. Can be restored by setting archived=false via experiment-update.

Reset (experiment-reset)

Returns an experiment to draft state. Clears start_date, end_date, conclusion, and archived.

• Preconditions: must not already be in draft
• Variants: flag is left unchanged — users continue seeing assigned variants
• Analysis: previously collected data still exists but won't be included in results unless start_date is adjusted after re-launch

No request body.

Duplicate (experiment-duplicate)

Creates a copy as a new draft with fresh dates and no results.

Important: always provide a unique feature_flag_key different from the original. If the same key is used, both experiments share a flag — changes to one affect both.

Optional: custom name (defaults to "Original Name (Copy)").

Decision framework

| Situation | Action | Tool | | -------------------------------------------------- | ------------------------ | ------------------------- | | Draft ready, flag implemented, metrics set | Launch | experiment-launch | | Clear winner, significant results | Ship the winning variant | experiment-ship-variant | | No significant difference after sufficient time | End as inconclusive | experiment-end | | Something wrong, need to stop exposure temporarily | Pause | experiment-pause | | Resume after pause | Resume | experiment-resume | | Experiment ended, ready to clean up | Archive | experiment-archive | | Need to start over with same config | Reset to draft | experiment-reset | | Want a similar experiment with a fresh start | Duplicate | experiment-duplicate |

Resolving experiments

All lifecycle actions require an experiment ID. If you don't have one, load the finding-experiments skill to resolve the user's reference (name, description, "latest", etc.) to a concrete ID before proceeding.

Error handling

| Error message | Meaning | | --------------------------------------- | ------------------------------------ | | "Experiment has already been launched." | Can't launch a non-draft experiment | | "Experiment has not been launched yet." | Can't end/pause/ship a draft | | "Experiment has already ended." | Can't end/pause a stopped experiment | | "Experiment is already paused." | Use resume instead | | "Experiment is not paused." | It's already active | | "Experiment is already in draft state." | Nothing to reset | | "Experiment is already archived." | Already done |

When you get a 400, explain the situation to the user rather than retrying.