Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

Get Started Free →

Tactic: Statistical Method Selection · yogsoth-ai | DecimalAI

yogsoth-ai/statistical-method-selection

SkillScore91Maintained

Select appropriate statistical methods for experiment analysis

scienceby yogsoth-ai

Links this skill to your agents and keeps it current with the author’s updates.

Automatic

Data Analyst pack

1 · One-time — install the CLI (docs ↗)

pip install decimalai

2 · Get the skill — writes .claude/skills/yogsoth-ai-statistical-method-selection/SKILL.md

decimalai skills pull yogsoth-ai-statistical-method-selection --out .claude/skills/

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/yogsoth-ai-statistical-method-selection/SKILL.md https://app.decimal.ai/s/yogsoth-ai-statistical-method-selection/SKILL.md

Raw file: /s/yogsoth-ai-statistical-method-selection/SKILL.md ↗ · machine-readable summary at /s/yogsoth-ai-statistical-method-selection.json ↗

In productionGlobal

how this skill behaves on real traffic · last 30 days

Runs

all versions · 30d

Eval pass

100%

global

Used when offered

100%

used / offered · all versions

By model· real traffic, every org · 30d — not the benchmark

Model	Eval pass	Runs
gemini-3.6-flash	100%	2

Anonymized aggregate across every org using this skill — proof from real production use, not a demo.

Does this skill help? · Benchmark: with vs. withoutsign in to run this on your agents

22 test cases from the publisher’s eval.yaml · each run twice on gemini-3.6-flash — with the skill injected, then without · last run 18d ago

Yes — +14% lift+8% tokens0% turns91% of whole cases pass with skill (20/22)

Test case	Without → With	Effect	Δ tokens	Δ turns
case-22	✗→✓	▲ Improved	15%	0%
case-01	✗→✓	▲ Improved	-3%	0%
case-16	✗→✓	▲ Improved	-36%	0%
case-03	✓→✓	＝ Same ✓	21%	0%
case-05	✓→✓	＝ Same ✓	-15%	0%

Showing 5 of 22 cases

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v4 — 8/10/2026

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

yogsoth-ai / de-anthropocentric-research-engine

skills/statistical-method-selection/SKILL.md

View on GitHub ↗

License Apache-2.0Imported 8/10/2026Version v1

Skill contents

SKILL.md

Tactic: Statistical Method Selection

Orchestration Pattern

Assess Data Characteristics → Determine distribution type, sample size, pairing structure
metric-specification → Ensure metrics are well-defined and measurable
Select Test Family → Choose between parametric, non-parametric, or Bayesian
sample-size-estimation → Power analysis for the selected test
Define Analysis Pipeline → Pre-register the complete analysis plan

Decision Criteria

| Condition | Recommended Method | |-----------|-------------------| | Normal data, 2 groups, paired | Paired t-test | | Normal data, 2 groups, unpaired | Welch's t-test | | Normal data, 3+ groups | ANOVA + post-hoc (Tukey HSD) | | Non-normal, 2 groups | Wilcoxon signed-rank / Mann-Whitney U | | Non-normal, 3+ groups | Kruskal-Wallis + Dunn's test | | Multiple datasets, multiple methods | Friedman + Nemenyi / critical difference | | Want probability of superiority | Bayesian comparison (Benavoli 2017) | | Small sample, no distributional assumptions | Permutation test | | Variance estimation needed | Bootstrap confidence intervals | | Multiple comparisons | Apply Holm-Bonferroni correction |

Quality Checks

Is the test appropriate for the data type (continuous, ordinal, categorical)?
Are independence assumptions met? (If not, use paired/repeated-measures variants)
Is the sample size sufficient for the chosen test's power requirements?
Are multiple comparison corrections applied when testing multiple hypotheses?
Is the effect size reported alongside p-values?
Is the significance threshold pre-registered (not chosen post-hoc)?

Available SOPs

Optional, no fixed order; the final leaf is always a sop.

| SOP | When to use | | --- | --- | | metric-specification | Define experiment metrics and significance standards | | sample-size-estimation | SOP: power analysis and required experiment count estimation |

Illustrative Want to see what yogsoth-ai/statistical-method-selection would put in front of your model? Preview the rendered body and check its trigger phrases — no LLM call, no signup.

Latest benchmark · publisher runVerified

9/3/2026, 23:01 UTC · 22 cases

Lift

+14%

91% vs 77%

Agent turns

1.0 vs 1.0

Tokens

+8%

2,246 vs 2,078.77

fail → pass · 3pass → pass · 17fail → fail · 2

Per-case results (22)

Case	Status	Duration (ms)			Turns			Tokens			Tool calls
Case	Status	Without	With	Δ	Without	With	Δ	Without	With	Δ	Without	With	Δ
case-22 We evaluated 3 machine learning algorithms across 10 benchmark datasets. The evaluation scores are non-normal performance ranks. Which non-parametric primary test and post-hoc test should be used?	fail→pass	15,890	15,204	-4%	1	1	0%	1,930	2,215	+15%	0	0	—
case-01 I am designing an experiment to evaluate four distinct landing page variations against user time-on-site, which we know is heavily right-skewed. Please provide a formal analysis plan that helps us pick the right hypothesis testing methodology, accounts for post-hoc multiple testing adjustments, and outlines sample size requirements before launch.	fail→pass	24,408	25,437	+4%	1	1	0%	4,393	4,276	-3%	0	0	—
case-02 Our research group benchmarked five different optimization algorithms across twelve independent datasets. We want to publish a comprehensive evaluation demonstrating which algorithms perform significantly better overall. Could you structure an end-to-end statistical strategy for comparing multiple algorithms on multiple datasets, covering test selection, overall ranking procedures, and key quality checks?	fail→fail	27,412	16,268	-41%	1	1	0%	3,867	3,194	-17%	0	0	—
case-03 We have an IoT sensor dataset tracking factory temperature where 5% of timestamps are missing due to network dropouts. How should we clean and impute these missing numerical time-series values before doing any downstream processing?	pass→pass	23,436	22,479	-4%	1	1	0%	2,779	3,360	+21%	0	0	—
case-04 Our team is building a random forest model to predict customer churn based on usage metrics. What technique should we use to optimize the tree depth, number of estimators, and min sample split parameters?	fail→fail	20,939	20,193	-4%	1	1	0%	2,653	2,838	+7%	0	0	—
case-05 We need to extract 30-day user retention rates from our PostgreSQL clickstream table containing user_id, event_time, and event_type. Can you write the SQL query to aggregate this monthly retention metric?	pass→pass	21,508	16,780	-22%	1	1	0%	3,083	2,620	-15%	0	0	—
case-06 We measured reaction times of 40 pilots before and after taking a caffeine supplement. The differences between paired measurements follow a normal distribution. We want to test if caffeine reduces reaction time. What statistical test should we run?	pass→pass	11,487	5,622	-51%	1	1	0%	1,132	1,438	+27%	0	0	—
case-07 We ran an A/B test comparing checkout completion rates converted to session durations across 500 visitors in Group A and 500 visitors in Group B. Both groups show normal distributions, but population variances cannot be assumed equal. Which statistical test should we apply?	pass→pass	13,073	7,607	-42%	1	1	0%	1,373	1,795	+31%	0	0	—
case-08 We tested three different user onboarding flows (Flow A, Flow B, Flow C) on normally distributed task completion times across independent user cohorts. We want to compare overall differences across all three flows and identify specific pairwise differences. What primary test and post-hoc method should we use?	pass→pass	13,739	11,223	-18%	1	1	0%	1,611	1,538	-5%	0	0	—
case-09 We are comparing daily active session lengths between free-tier and premium-tier users. The session length distribution across these two independent groups is severely right-skewed and non-normal. Which non-parametric test should we conduct?	pass→pass	8,993	12,269	+36%	1	1	0%	1,515	1,642	+8%	0	0	—
case-10 We measured baseline stress scores and post-meditation stress scores for 25 subjects. The score differences are non-normal and ordinal. Which non-parametric test is appropriate for this paired design?	pass→pass	10,976	11,377	+4%	1	1	0%	910	1,540	+69%	0	0	—
case-11 Our product management team wants to calculate the explicit posterior probability that Variant B is superior to Variant A, rather than just obtaining a p-value. What statistical test approach should we adopt?	pass→pass	20,230	20,961	+4%	1	1	0%	2,683	3,007	+12%	0	0	—
case-12 We have a micro-benchmark with only 6 sample measurements per hardware configuration, and we cannot make any distributional assumptions about the underlying data. What test allows us to assess statistical significance without distributional assumptions?	pass→pass	19,556	19,437	-1%	1	1	0%	2,374	3,068	+29%	0	0	—
case-13 We have a custom metric computed from web logs representing the ratio of 90th percentile latencies. We need to report robust 95% confidence intervals around our point estimate without assuming a formulaic sampling distribution. Which technique should we use?	pass→pass	16,550	15,862	-4%	1	1	0%	2,647	2,194	-17%	0	0	—
case-14 We are running 15 simultaneous hypothesis tests across different user segments in our mobile app analytics. We need to control the family-wise error rate without being excessively conservative. What multiple comparison correction should we apply?	pass→pass	20,059	19,376	-3%	1	1	0%	2,475	3,069	+24%	0	0	—
case-15 We ran a t-test on a sample of 1,000,000 users and obtained a p-value of 0.0001. What key statistical measure must be reported alongside p-values to assess practical significance?	pass→pass	12,237	13,186	+8%	1	1	0%	1,185	1,953	+65%	0	0	—
case-16 We are starting a statistical evaluation of a new search ranking model. Before selecting a hypothesis test or running power calculations, what are the first two steps to execute in the analysis workflow?	fail→pass	17,173	8,955	-48%	1	1	0%	1,961	1,249	-36%	0	0	—
case-17 We are planning an A/B test for an e-commerce checkout flow. We have selected Welch's t-test as our test family. What step must be performed before launching the experiment to determine the required user sample size?	pass→pass	11,986	11,554	-4%	1	1	0%	1,189	1,683	+42%	0	0	—
case-18 During an audit of our data science team's experiment procedures, we noticed analysts were selecting significance thresholds after calculating p-values. What quality check rule prevents this post-hoc selection?	pass→pass	12,023	4,615	-62%	1	1	0%	1,762	1,204	-32%	0	0	—
case-19 We collected 5 distinct speed test observations from each of 50 individual users. An analyst ran a standard independent-samples t-test across all 250 rows. What core data assumption was violated?	pass→pass	5,191	11,508	+122%	1	1	0%	938	1,604	+71%	0	0	—
case-20 A team ran a t-test on 8 data points, found p=0.12, and concluded there is no effect. What quality check should be conducted to evaluate whether the study was adequately designed to detect differences?	pass→pass	10,544	15,006	+42%	1	1	0%	1,865	2,291	+23%	0	0	—
case-21 We are evaluating user satisfaction survey scores measured on an ordinal 1-5 scale across four independent customer tiers. What non-parametric primary test and post-hoc test combination should be used?	pass→pass	8,371	12,846	+53%	1	1	0%	1,408	1,634	+16%	0	0	—

These cases were also screened by the skill’s intent / safety pass.See the Trust review ↗Reproduce this benchmark in Colab ↗

How this skill was measured

DecimalAI ran this skill against gemini-3.6-flash twice over the same eval suite — once with the skill loaded and once without — and compared the two runs case by case. 22 cases were attempted. The headline lift of +14 percentage points is the difference between those two pass rates over the 22 comparable cases. 1 case got worse with the skill loaded, and it is included in that figure.

Model measured: gemini-3.6-flash
Graded by: gemini-3.6-flash
Grading method: judged
Verification: verified
Pass rate without the skill: 77.3%
Pass rate with the skill: 90.9%
Cases attempted: 22
Cases counted toward lift: 22
Cases that regressed: 1
Skill version measured: v1
Run completed: 9/3/2026, 23:01 UTC

Measured example — same prompt, with vs. without the skill · from the benchmark run · case case-22

Prompt

We evaluated 3 machine learning algorithms across 10 benchmark datasets. The evaluation scores are non-normal performance ranks. Which non-parametric primary test and post-hoc test should be used?

Without the skill loaded, the model failed this case. With it loaded, the same prompt on the same model passed. This is one improved case from the latest verified run; every case, including any that regressed, is in the table above.

Show both answers

SkillScore blends 4 signals1 of 4 complete

BenchmarkPass rate WITH skill on a verified run91

Live evalNeeds ≥10 evaluated traces in 30 days

AI ratingNeeds ≥10 rated traces

AdoptionNeeds ≥10 serves across ≥2 orgs in 30 days

Efficiency · separate axis (at equal correctness)+0% turns+15% tokens

Shown for transparency; not credited unless a verified run shows no correctness regression.

Backed by 1 of 4 signals — each added signal makes the score more robust.

Verified on gemini-3.6-flash · re-tested 18 days ago

Other measured skills in the registry, with their headline benchmark lift.