Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

A/B Test Setup · alirezarezvani | DecimalAI

databy alirezarezvani

A/B Test Setup

alirezarezvani/ab-test-setup

Source alirezarezvani/claude-skills ↗License MIT

When the user wants to plan, design, or implement an A/B test or experiment. Also use when the user mentions "A/B test," "split test," "experiment," "test this change," "variant copy," "multivariate test," "hypothesis," "conversion experiment," "statistical significance," or "test this." For tracking implementation, see analytics-tracking.

SkillScore86Maintained

Automatic

Data Analyst pack

Last verified 8/4/2026

What it changes

Passed 19 of 22 test cases with the skill, and 16 of 22 without it.

Without skill73%

With skill86%

Checks it passed that the model alone missed

Identifies PostHog as a server-side experimentation platform.
Includes the --daily-traffic flag set to 2000 in the sample size script command.
Specifies calling python3 scripts/sample_size_calculator.py with baseline 0.05 and MDE 0.20.

3 fixed16 passed either way3 still failing0 broke

Pass-rate lift: +14 pp
Tokens: +89%
Agent turns: 0%
Response time: -21%

Model gemini-3.6-flashMeasured 8/4/2026v119 of 22 cases passed with skill

Tokens, turns and time describe resource use, not a dollar price.

Use this skill

Save it into your project's skills folder. Claude Code picks it up in its next session.

curl -fsSL --create-dirs -o .claude/skills/alirezarezvani-ab-test-setup/SKILL.md https://app.decimal.ai/s/alirezarezvani-ab-test-setup/SKILL.md

Want it pinned and updatable? pip install "decimalai>=0.15", then decimalai skills add alirezarezvani-ab-test-setup, and start Claude Code with claude --plugin-dir "$(git rev-parse --show-toplevel)/.agents"

1 · One-time — install the CLI (docs ↗)

pip install "decimalai>=0.15"

2 · Get the skill — writes .agents/skills/alirezarezvani-ab-test-setup/ and pins it in .decimal/skills.lock

decimalai skills add alirezarezvani-ab-test-setup

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/alirezarezvani-ab-test-setup/SKILL.md https://app.decimal.ai/s/alirezarezvani-ab-test-setup/SKILL.md

Raw file: /s/alirezarezvani-ab-test-setup/SKILL.md ↗ · machine-readable summary at /s/alirezarezvani-ab-test-setup.json ↗

Safety checks: 3 of 3 passed

Code scan CleanIntent PassedContent Clean

See it in action

One of 3 test cases the skill fixed. Same model, same prompt, with and without the skill.

1 of 3

Prompt

When running the sample size calculator script for a landing page receiving 2,000 daily visitors, which flag provides the estimated duration in days?

What was checked, without and with the skill
Checked	Without skill	With skill
Includes the --daily-traffic flag set to 2000 in the sample size script command.	Missed	Passed

case-04 · gemini-3.6-flash · 8/4/2026 · answers are capped and redacted

What's inside

The instructions your model reads when the skill loads.

SKILL.md305 linessingle file

A/B Test Setup

You are an expert in experimentation and A/B testing. Your goal is to help design tests that produce statistically valid, actionable results.

Initial Assessment

Check for product marketing context first: If .claude/product-marketing-context.md exists, read it before asking questions. Use that context and only ask for information not already covered or specific to this task.

Before designing a test, understand:

Test Context - What are you trying to improve? What change are you considering?
Current State - Baseline conversion rate? Current traffic volume?
Constraints - Technical complexity? Timeline? Tools available?

Core Principles

1. Start with a Hypothesis

Not just "let's see what happens"
Specific prediction of outcome
Based on reasoning or data

2. Test One Thing

Single variable per test
Otherwise you don't know what worked

3. Statistical Rigor

Pre-determine sample size
Don't peek and stop early
Commit to the methodology

4. Measure What Matters

Primary metric tied to business value
Secondary metrics for context
Guardrail metrics to prevent harm

Track record

Benchmark

19 / 22

test cases passed, vs 16 / 22 without

Adoption

Users: 0
Times used: 0

Safety and source details

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v4 — 8/3/2026

View 15 scanner checks

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

alirezarezvani / claude-skills

.gemini/skills/ab-test-setup/SKILL.md

View on GitHub ↗

License MITImported 8/3/2026Version v1

Other skills for the same job

Published skills whose descriptions closely match this one, with their headline benchmark lift.

More skills from the same source and nearby in the registry, with their headline benchmark lift.

Ready to supercharge your AI agents?

See it in action

What's inside

A/B Test Setup

Initial Assessment

Core Principles

1. Start with a Hypothesis

2. Test One Thing

3. Statistical Rigor

4. Measure What Matters

Track record

Benchmark

Adoption

SkillSafety

Source

Other skills for the same job

Related skills