Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →rampstackco/data-warehouse-experimentation
Running experiments out of the data warehouse instead of via dedicated experiment platforms. SQL-based assignment, exposure logging discipline, metric definitions in dbt models, statistical analysis in SQL or Python, variance reduction with CUPED, sequential testing, and the operational tradeoffs vs platforms like Statsig and Optimizely.
Passed 24 of 24 test cases with the skill, and 21 of 24 without it.
3 fixed21 passed either way0 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 3 test cases the skill fixed. Same model, same prompt, with and without the skill.
An analyst is writing a BigQuery SQL query to deterministically assign users into control and treatment groups for a homepage button experiment without using an external feature flagging service. They are considering using RAND() to split users 50/50. Provide the standard BigQuery SQL function pattern for deterministic user bucketing that prevents assignment collision across different experiments.
| Checked | Without skill | With skill |
|---|---|---|
| Recommends FARM_FINGERPRINT using user_id concatenated with a salt string. | Passed | Passed |
| Uses MOD(ABS(...), 100) arithmetic for allocation. | Missed | Passed |
The instructions your model reads when the skill loads.
A senior data scientist's playbook for running experiments natively out of BigQuery, Snowflake, or any modern data warehouse, with metric definitions in dbt and statistical analysis in SQL or Python.
Most companies that run experiments at scale use a dedicated platform. Statsig, Optimizely, LaunchDarkly with experimentation, PostHog, Amplitude Experiment. The platforms are good. They handle assignment, instrumentation, and analysis in one product, and the SQL-savvy data team does not have to reinvent the variance reduction wheel.
There is a different operational model that mature data teams increasingly choose: warehouse-native experimentation. Assignment happens in code or via feature flags. Exposure events fire to the warehouse like any other event. Metrics are defined as dbt models. Statistical analysis runs as SQL or in a Python notebook against warehouse data. The "experiment platform" is just your existing data stack.
This skill covers when warehouse-native is the right call, the architecture, and the specific techniques that make it work: assignment patterns, exposure logging discipline, metric definitions in dbt, t-tests and CUPED in SQL, sequential testing, and the pitfalls that take down homegrown setups.
When to use this skill: deciding between platform vs warehouse-native, building a warehouse-native experiment infrastructure, auditing an existing one, or running a specific experiment when the platform of record cannot handle a custom metric or segmentation.
This skill spans the operational execution model for warehouse-native experimentation. It does not replace the methodology and interpretation skills; it composes with them.
experiment-design covers methodology: hypotheses, sample size, randomization unit, primary metric. Tool-agnostic. Read it first to design the experiment correctly regardless of where it runs.experimentation-analytics covers interpretation: confidence intervals, p-values, effect size, decision frameworks. Tool-agnostic. Read it when results land.experimentation-platform-orchestrator covers the platform-vs-warehouse decision in detail. Read it to decide whether to use a platform or this skill.feature-flagging covers assignment infrastructure when not running through a platform. Read it for the flag-management discipline that this skill assumes.data-warehouse-experimentation) covers the operational execution: SQL-based assignment, exposure logging, metric definitions in dbt, statistical analysis in SQL or Python, variance reduction, sequential testing.The distinction is between "what to do" (the methodology and interpretation skills) and "how to do it without a vendor platform" (this skill). Read this skill after you have decided warehouse-native is the right call. If you are still deciding, start with experimentation-platform-orchestrator.
Six factors push the decision toward warehouse-native.
Five factors push toward platform.
test cases passed, vs 21 / 24 without
More skills from the same source and nearby in the registry, with their headline benchmark lift.