Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

Get Started Free →

A/B Test Setup · minicoohei | DecimalAI

marketingby minicoohei

A/B Test Setup

minicoohei/ab-test-setup

Source minicoohei/ai-agent-camp ↗License Not recorded

A/Bテストや実験の設計・実装を支援するスキル。「A/Bテストを設計して」「スプリットテストしたい」「仮説を立ててテストしたい」「バリアントを比較」等のリクエストで発動。トラッキング実装は analytics-tracking を参照。

SkillScore88Maintained

Automatic

Product Management pack

Last verified 8/21/2026

What it changes

Passed 22 of 25 test cases with the skill, and 17 of 25 without it.

Without skill68%

With skill88%

Checks it passed that the model alone missed

Specify approximately 7,000 visitors per variant for a 5% baseline with 20% relative lift.
Recommend approximately 3,000 visitors per variant for a 10% baseline with 20% relative lift.
Prompt for baseline conversion rate or current traffic volume context to determine required test duration.

and 3 more in the full report

6 fixed16 passed either way2 still failing1 broke

Pass-rate lift: +20 pp
Tokens: +62%
Agent turns: 0%
Response time: -20%

Model gemini-3.6-flashMeasured 8/21/2026v122 of 25 cases passed with skill

Tokens, turns and time describe resource use, not a dollar price.

Use this skill

Save it into your project's skills folder. Claude Code picks it up in its next session.

curl -fsSL --create-dirs -o .claude/skills/minicoohei-ab-test-setup/SKILL.md https://app.decimal.ai/s/minicoohei-ab-test-setup/SKILL.md

Want it pinned and updatable? pip install "decimalai>=0.15", then decimalai skills add minicoohei-ab-test-setup, and start Claude Code with claude --plugin-dir "$(git rev-parse --show-toplevel)/.agents"

1 · One-time — install the CLI (docs ↗)

pip install "decimalai>=0.15"

2 · Get the skill — writes .agents/skills/minicoohei-ab-test-setup/ and pins it in .decimal/skills.lock

decimalai skills add minicoohei-ab-test-setup

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/minicoohei-ab-test-setup/SKILL.md https://app.decimal.ai/s/minicoohei-ab-test-setup/SKILL.md

Raw file: /s/minicoohei-ab-test-setup/SKILL.md ↗ · machine-readable summary at /s/minicoohei-ab-test-setup.json ↗

Safety checks: 3 of 3 passed

Code scan CleanIntent PassedContent Clean

See it in action

One of 6 test cases the skill fixed. Same model, same prompt, with and without the skill.

1 of 6

Prompt

Before we begin designing an experiment for our onboarding wizard, what project context file should be checked if present in the codebase workspace?

What was checked, without and with the skill
Checked	Without skill	With skill
Reference checking `.claude/product-marketing-context.md` for existing product context before asking redundant questions.	Missed	Passed

case-18 · gemini-3.6-flash · 8/21/2026 · answers are capped and redacted

What's inside

The instructions your model reads when the skill loads.

SKILL.md259 lines3 files

A/B Test Setup

You are an expert in experimentation and A/B testing. Your goal is to help design tests that produce statistically valid, actionable results.

Initial Assessment

Check for product marketing context first: If .claude/product-marketing-context.md exists, read it before asking questions. Use that context and only ask for information not already covered or specific to this task.

Before designing a test, understand:

Test Context - What are you trying to improve? What change are you considering?
Current State - Baseline conversion rate? Current traffic volume?
Constraints - Technical complexity? Timeline? Tools available?

Core Principles

1. Start with a Hypothesis

Not just "let's see what happens"
Specific prediction of outcome
Based on reasoning or data

2. Test One Thing

Single variable per test
Otherwise you don't know what worked

3. Statistical Rigor

Pre-determine sample size
Don't peek and stop early
Commit to the methodology

4. Measure What Matters

Primary metric tied to business value
Secondary metrics for context
Guardrail metrics to prevent harm

Track record

Benchmark

22 / 25

test cases passed, vs 17 / 25 without

Adoption

Users: 0
Times used: 0

Safety and source details

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v4 — 8/3/2026

View 15 scanner checks

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

minicoohei / ai-agent-camp

skills/ab-test-setup/SKILL.md

View on GitHub ↗

Imported 8/3/2026Version v1Bundled 2 files

Other skills for the same job

Published skills whose descriptions closely match this one, with their headline benchmark lift.

More skills from the same source and nearby in the registry, with their headline benchmark lift.

Ready to supercharge your AI agents?

See it in action

What's inside

A/B Test Setup

Initial Assessment

Core Principles

1. Start with a Hypothesis

2. Test One Thing

3. Statistical Rigor

4. Measure What Matters

Track record

Benchmark

Adoption

SkillSafety

Source

Other skills for the same job

Related skills