Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

Get Started Free →

A/B Test Design Skill · a5c-ai | DecimalAI

a5c-ai/A-B Test Design

NewSkillScoreMaintained

Statistical experiment design and analysis capabilities for product experimentation

scienceby a5c-ai

Links this skill to your agents and keeps it current with the author’s updates.

Automatic

Data Analyst pack

1 · One-time — install the CLI (docs ↗)

pip install decimalai

2 · Get the skill — writes .claude/skills/a5c-ai-a-b-test-design/SKILL.md

decimalai skills pull a5c-ai-a-b-test-design --out .claude/skills/

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/a5c-ai-a-b-test-design/SKILL.md https://app.decimal.ai/s/a5c-ai-a-b-test-design/SKILL.md

Raw file: /s/a5c-ai-a-b-test-design/SKILL.md ↗ · machine-readable summary at /s/a5c-ai-a-b-test-design.json ↗

In productionGlobal

how this skill behaves on real traffic · last 30 days

No production data for this skill yet — usage numbers appear here once agents start running it.

Anonymized aggregate across every org using this skill — proof from real production use, not a demo.

Does this skill help? · Benchmark: with vs. withoutsign in to run this on your agents

24 test cases from the publisher’s eval.yaml · each run twice on gemini-3.6-flash — with the skill injected, then without · last run 29d ago

No measurable change+35% tokens0% turns17% of whole cases pass with skill (4/24)

Test case	Without → With	Effect	Δ tokens	Δ turns
case-21	✓→✓	＝ Same ✓	45%	0%
case-22	✓→✓	＝ Same ✓	30%	0%
case-23	✓→✓	＝ Same ✓	130%	0%
case-24	✓→✓	＝ Same ✓	28%	0%
case-01	✗→✗	＝ Same ✗	-14%	0%

Showing 5 of 24 cases

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v4 — 8/10/2026

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

a5c-ai / babysitter

library/specializations/product-management/skills/ab-test-design/SKILL.md

View on GitHub ↗

License MITImported 8/10/2026Version v1

Skill contents

SKILL.md

A/B Test Design Skill

Overview

Specialized skill for statistical experiment design and analysis capabilities. Enables product teams to design rigorous experiments, calculate sample sizes, and interpret results with statistical confidence.

Capabilities

Experiment Design

Calculate required sample sizes for experiments
Design experiment variants and hypotheses
Define success metrics and guardrail metrics
Create experiment documentation templates
Design multi-variant tests (A/B/n)
Plan sequential and Bayesian experiments

Statistical Analysis

Validate statistical significance of results
Calculate practical significance and effect sizes
Detect interaction effects and segments
Perform power analysis
Calculate confidence intervals
Handle multiple comparison corrections

Decision Support

Recommend ship/iterate/kill decisions
Identify segment-specific impacts
Assess long-term vs short-term effects
Generate experiment reports
Track experiment velocity metrics

Target Processes

This skill integrates with the following processes:

product-market-fit.js - Validation experiments for PMF hypotheses
conversion-funnel-analysis.js - Funnel optimization experiments
beta-program.js - A/B testing during beta phases

Input Schema

json
{
  "type": "object",
  "properties": {
    "experimentType": {
      "type": "string",
      "enum": ["ab", "multivariate", "sequential", "bandit"],
      "description": "Type of experiment to design"
    },
    "hypothesis": {
      "type": "string",
      "description": "Hypothesis to test"
    },
    "primaryMetric": {
      "type": "object",
      "properties": {
        "name": { "type": "string" },
        "baseline": { "type": "number" },
        "mde": { "type": "number", "description": "Minimum detectable effect" }
      }
    },
    "guardrailMetrics": {
      "type": "array",
      "items": { "type": "string" },
      "description": "Metrics that should not regress"
    },
    "trafficAllocation": {
      "type": "number",
      "description": "Percentage of traffic for experiment"
    },
    "confidenceLevel": {
      "type": "number",
      "default": 0.95,
      "description": "Statistical confidence level"
    }
  },
  "required": ["experimentType", "hypothesis", "primaryMetric"]
}

Output Schema

json
{
  "type": "object",
  "properties": {
    "experimentPlan": {
      "type": "object",
      "properties": {
        "name": { "type": "string" },
        "hypothesis": { "type": "string" },
        "variants": { "type": "array", "items": { "type": "object" } },
        "sampleSize": { "type": "number" },
        "duration": { "type": "string" },
        "metrics": { "type": "object" }
      }
    },
    "powerAnalysis": {
      "type": "object",
      "properties": {
        "requiredSampleSize": { "type": "number" },
        "estimatedDuration": { "type": "string" },
        "power": { "type": "number" }
      }
    },
    "implementation": {
      "type": "object",
      "properties": {
        "trackingEvents": { "type": "array", "items": { "type": "string" } },
        "segmentation": { "type": "array", "items": { "type": "string" } },
        "rolloutPlan": { "type": "string" }
      }
    },
    "analysisFramework": {
      "type": "object",
      "properties": {
        "primaryAnalysis": { "type": "string" },
        "secondaryAnalyses": { "type": "array", "items": { "type": "string" } },
        "decisionCriteria": { "type": "object" }
      }
    }
  }
}

Usage Example

javascript
const experimentDesign = await executeSkill('ab-test-design', {
  experimentType: 'ab',
  hypothesis: 'Adding social proof to pricing page increases conversion by 10%',
  primaryMetric: {
    name: 'pricing_page_conversion',
    baseline: 0.05,
    mde: 0.10
  },
  guardrailMetrics: ['revenue_per_visitor', 'bounce_rate'],
  trafficAllocation: 50,
  confidenceLevel: 0.95
});

Dependencies

Statistical libraries for power analysis
Experimentation platform integrations (Optimizely, LaunchDarkly, etc.)

Illustrative Want to see what a5c-ai/A-B Test Design would put in front of your model? Preview the rendered body and check its trigger phrases — no LLM call, no signup.

Latest benchmark · publisher runVerified

8/29/2026, 21:41 UTC · 24 cases

Lift

0%

17% vs 17%

Agent turns

0%

1.0 vs 1.0

Tokens

+35%

4,404.17 vs 3,257.88

pass → pass · 4fail → fail · 20

Per-case results (24)

Case	Status	Duration (ms)			Turns			Tokens			Tool calls
Case	Status	Without	With	Δ	Without	With	Δ	Without	With	Δ	Without	With	Δ
case-01 We are setting up a standard A/B test to evaluate whether adding customer testimonials on our checkout page increases completed purchases. Our baseline purchase rate is 0.04, and we want to detect a minimum detectable effect of 10% relative uplift at a 95% confidence level, allocating 50% of incoming traffic. Please keep refund rate and site bounce rate as guardrail metrics. Produce a complete response containing the experiment plan, statistical power analysis, implementation tracking requirements, and an analysis framework with decision criteria.	fail→fail	40,093	24,161	-40%	1	1	0%	6,640	5,694	-14%	0	0	—
case-02 I need to design a sequential experiment testing the hypothesis that changing our onboarding flow from four steps to two steps improves 7-day retention. Our current baseline retention is 0.22, and we are aiming to detect an MDE of 0.08 with 95% confidence using 100% traffic allocation. Customer support ticket volume and session length should serve as guardrail metrics. Please return an output detailing the experiment plan, power analysis, implementation setup including tracking events, and the analysis framework.	fail→fail	49,744	25,231	-49%	1	1	0%	8,295	5,081	-39%	0	0	—
case-03 Our team wants to launch a multivariate experiment to compare three distinct pricing table designs aimed at increasing annual subscription sign-ups. Our baseline conversion is 0.15, the minimum detectable effect is 0.05, and we plan to allocate 80% of eligible web traffic with a 0.95 confidence level while monitoring overall churn as a guardrail metric. Generate a structured outcome that includes the overall test plan, required sample size power analysis, implementation and segmentation steps, and the decision criteria analysis framework.	fail→fail	41,346	19,651	-52%	1	1	0%	8,246	4,904	-41%	0	0	—
case-04 Our web team wants to test changing the primary checkout CTA button from blue to green on our main e-commerce site to boost order completions. Baseline conversion is 0.03, MDE is 0.05, traffic allocation is 50%, and confidence level is 0.95, with cart abandonment as a guardrail metric. Summarize the output in a nice Markdown document with headings for executive readability.	fail→fail	15,416	27,730	+80%	1	1	0%	2,963	5,555	+87%	0	0	—
case-05 We want to set up a multi-armed bandit experiment on our news portal homepage hero banner to automatically route traffic to the top-performing headline among 4 variants. Baseline click-through rate is 0.08, MDE is 0.12, traffic allocation is 100%, confidence level is 0.95, and page bounce rate is the guardrail metric. Feel free to just outline the concept in plain conversational paragraphs without rigid JSON structures.	fail→fail	12,676	14,024	+11%	1	1	0%	2,067	3,198	+55%	0	0	—
case-06 Design a multivariate test for our SaaS landing page, testing 2 headlines x 2 hero images (4 total variants) to increase sign-up conversion from a baseline of 0.06 with an MDE of 0.15. Keep unsubscribe rate as a guardrail metric and use 100% traffic allocation at 95% confidence. Write a narrative summary of your recommendation instead of structured JSON fields.	fail→fail	22,513	27,960	+24%	1	1	0%	3,800	6,299	+66%	0	0	—
case-07 We are planning a sequential testing design for a cart upsell modal on our mobile app to detect whether showing personalized add-on recommendations increases average order value. Baseline conversion is 0.10, MDE is 0.10, traffic allocation is 50%, confidence level is 0.95, and order cancellation rate is a guardrail metric. Can you just provide a quick summary without including sample size calculations or power analysis?	fail→fail	4,592	9,244	+101%	1	1	0%	877	2,822	+222%	0	0	—
case-08 We want to evaluate a new search ranking model on our marketplace app to increase search-to-purchase conversion. Baseline conversion is 0.12, target MDE is 0.05, traffic allocation is 20%, and confidence level is 0.99. Zero-result search rate and latency are guardrails. Please provide a brief text memo outlining the design.	fail→fail	12,004	15,329	+28%	1	1	0%	2,302	4,112	+79%	0	0	—
case-09 Our marketing group wants to run an A/B experiment testing personalized subject lines vs generic subject lines for our weekly digest email. Baseline open rate is 0.20, MDE is 0.10, traffic allocation is 100%, and confidence level is 0.95. Unsubscribe rate is a guardrail metric. Output a bulleted list in Markdown for the team slack channel.	fail→fail	12,454	11,864	-5%	1	1	0%	2,024	3,630	+79%	0	0	—
case-10 We want to run a multi-armed bandit test dynamically allocating traffic across three pricing layout options for our digital subscription platform. Baseline conversion is 0.05, MDE is 0.10, traffic allocation is 50%, confidence level is 0.95, and refund rate is a guardrail metric. Just write a summary email draft to the product manager.	fail→fail	7,476	8,027	+7%	1	1	0%	1,356	2,600	+92%	0	0	—
case-11 We are setting up a multivariate experiment testing 3 payment icon positions by 2 coupon code field placements during checkout. Baseline checkout completion is 0.35, MDE is 0.04, traffic allocation is 100%, confidence level is 0.95, and payment failure rate is a guardrail metric. Produce a standard Markdown document listing the plan.	fail→fail	18,960	25,459	+34%	1	1	0%	3,493	6,193	+77%	0	0	—
case-12 Design a sequential testing experiment for a paywall modal triggered after 3 article views on our publishing website. Baseline subscription conversion is 0.02, MDE is 0.20, traffic allocation is 30%, confidence level is 0.95, and user churn is a guardrail metric. Provide the design as a plain narrative text response.	fail→fail	16,863	18,159	+8%	1	1	0%	3,179	4,544	+43%	0	0	—
case-13 We are evaluating an A/B experiment testing a 7-day free trial versus a 14-day free trial for our B2B productivity app. Baseline trial-to-paid conversion is 0.08, MDE is 0.15, traffic allocation is 50%, confidence level is 0.95, and trial sign-up drop-off is a guardrail metric. Create a Markdown table detailing the experiment.	fail→fail	14,295	16,655	+17%	1	1	0%	2,584	4,076	+58%	0	0	—
case-14 Our growth team wants to test adding a progress bar and completion badges to our user onboarding flow to boost week-1 activity. Baseline retention is 0.25, target MDE is 0.08, traffic allocation is 50%, and confidence level is 0.95. Support ticket volume is a guardrail metric. Provide a brief Markdown response with bullet points.	fail→fail	12,693	8,911	-30%	1	1	0%	1,852	2,883	+56%	0	0	—
case-15 We want to test 2 image aspect ratios against 2 CTA copy options on product cards across our mobile e-commerce catalog. Baseline click-through rate is 0.14, MDE is 0.07, traffic allocation is 100%, confidence level is 0.95, and return rate is a guardrail metric. Output the recommendation in bulleted Markdown text.	fail→fail	16,155	14,005	-13%	1	1	0%	3,066	3,783	+23%	0	0	—
case-16 We plan to deploy a contextual bandit experiment for our streaming platform's recommended watch next carousel. Baseline engagement rate is 0.18, MDE is 0.06, traffic allocation is 100%, confidence level is 0.95, and video buffering rate is a guardrail metric. Return a conversational chat message summarizing the plan.	fail→fail	5,037	13,872	+175%	1	1	0%	953	3,704	+289%	0	0	—
case-17 We are designing a sequential test for push notification delivery times (morning vs evening) for our fitness tracking application to increase daily active logging. Baseline daily logging is 0.30, MDE is 0.05, traffic allocation is 40%, confidence level is 0.95, and notification opt-out rate is a guardrail metric. Deliver the result as a text memorandum.	fail→fail	28,507	33,748	+18%	1	1	0%	4,525	7,397	+63%	0	0	—
case-18 We want to run an A/B experiment testing $10 cash credit vs $15 account credit for our peer referral program. Baseline referral completion rate is 0.025, MDE is 0.20, traffic allocation is 100%, confidence level is 0.95, and fraud rate is a guardrail metric. Summarize the experimental setup in a Markdown report.	fail→fail	15,261	28,701	+88%	1	1	0%	2,823	5,414	+92%	0	0	—
case-19 Our desktop app team wants to test making dark mode the default theme vs light mode default to measure 30-day user retention. Baseline 30-day retention is 0.40, target MDE is 0.05, traffic allocation is 50%, confidence level is 0.95, and app crash rate is a guardrail metric. Write down a plain text proposal.	fail→fail	14,312	14,326	+0%	1	1	0%	2,783	3,864	+39%	0	0	—
case-20 Design a multivariate test comparing 2 shipping speed displays by 2 free-shipping progress bar designs on our retail checkout page. Baseline conversion is 0.09, MDE is 0.10, traffic allocation is 80%, confidence level is 0.95, and shipping cost margin is a guardrail metric. Return a formatted Markdown file.	fail→fail	34,340	40,028	+17%	1	1	0%	6,481	8,362	+29%	0	0	—
case-21 We need a PostgreSQL query to pull historical daily active user conversion rates for users onboarded during Q1 2024 from our analytics database table analytics.user_events. Group the results by signup_cohort week and calculate the 7-day conversion percentage.	pass→pass	17,753	12,533	-29%	1	1	0%	2,584	3,746	+45%	0	0	—
case-22 Write a Node.js Express route handler that checks an existing feature flag key new-checkout-flow using a standard feature flagging SDK to serve either the new checkout component or legacy checkout component to an incoming request.	pass→pass	12,752	11,411	-11%	1	1	0%	2,431	3,171	+30%	0	0	—
case-23 We completed a user survey across 500 respondents regarding app satisfaction. We have a list of numerical ratings from 1 to 5. Please calculate the mean value for this single sample array: [4, 5, 3, 2, 4, 5, 1, 4, 3, 5, 4, 2, 5, 4, 3].	pass→pass	4,128	5,124	+24%	1	1	0%	979	2,254	+130%	0	0	—
case-24 We ran a funnel analysis on our sign-up flow table signup_funnel_logs and found that 10,000 users visited step 1, 4,500 reached step 2, and 1,800 completed step 3. Calculate the step-by-step conversion rates and overall drop-off rate for this completed funnel.	pass→pass	8,302	6,537	-21%	1	1	0%	1,886	2,414	+28%	0	0	—

These cases were also screened by the skill’s intent / safety pass.See the Trust review ↗Reproduce this benchmark in Colab ↗

How this skill was measured

DecimalAI ran this skill against gemini-3.6-flash twice over the same eval suite — once with the skill loaded and once without — and compared the two runs case by case. 24 cases were attempted. The headline lift of 0 percentage points is the difference between those two pass rates over the 24 comparable cases. 1 case got worse with the skill loaded, and it is included in that figure.

Model measured: gemini-3.6-flash
Graded by: gemini-3.6-flash
Grading method: judged
Verification: verified
Pass rate without the skill: 16.7%
Pass rate with the skill: 16.7%
Cases attempted: 24
Cases counted toward lift: 24
Cases that regressed: 1
Skill version measured: v1
Run completed: 8/29/2026, 21:41 UTC

SkillScore blends 4 signalsno signals yet

BenchmarkRun a verified benchmark with positive lift

Live evalNeeds ≥10 evaluated traces in 30 days

AI ratingNeeds ≥10 rated traces

AdoptionNeeds ≥10 serves across ≥2 orgs in 30 days

Efficiency · separate axis (at equal correctness)+0% turns+47% tokens

Shown for transparency; not credited unless a verified run shows no correctness regression.

No signals yet — run a benchmark or collect evaluated traces to earn a score.

Verified on gemini-3.6-flash · re-tested 29 days ago

Other measured skills in the registry, with their headline benchmark lift.