Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

Get Started Free →

Strategy: Result Analysis · yogsoth-ai | DecimalAI

yogsoth-ai/result-analysis

SkillScore75Maintained

Statistically analyze collected results, verify reproducibility, and synthesize findings

scienceby yogsoth-ai

Links this skill to your agents and keeps it current with the author’s updates.

Automatic

Data Analyst pack

1 · One-time — install the CLI (docs ↗)

pip install decimalai

2 · Get the skill — writes .claude/skills/yogsoth-ai-result-analysis/SKILL.md

decimalai skills pull yogsoth-ai-result-analysis --out .claude/skills/

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/yogsoth-ai-result-analysis/SKILL.md https://app.decimal.ai/s/yogsoth-ai-result-analysis/SKILL.md

Raw file: /s/yogsoth-ai-result-analysis/SKILL.md ↗ · machine-readable summary at /s/yogsoth-ai-result-analysis.json ↗

In productionGlobal

how this skill behaves on real traffic · last 30 days

No production data for this skill yet — usage numbers appear here once agents start running it.

Anonymized aggregate across every org using this skill — proof from real production use, not a demo.

Does this skill help? · Benchmark: with vs. withoutsign in to run this on your agents

24 test cases from the publisher’s eval.yaml · each run twice on gemini-3.6-flash — with the skill injected, then without · last run 18d ago

Yes — +42% lift+3% tokens0% turns75% of whole cases pass with skill (18/24)

Test case	Without → With	Effect	Δ tokens	Δ turns
case-01	✗→✓	▲ Improved	-28%	0%
case-04	✗→✓	▲ Improved	78%	0%
case-07	✗→✓	▲ Improved	68%	0%
case-14	✗→✓	▲ Improved	-33%	0%
case-11	✗→✓	▲ Improved	45%	0%

Showing 5 of 24 cases

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v4 — 8/10/2026

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

yogsoth-ai / de-anthropocentric-research-engine

skills/result-analysis/SKILL.md

View on GitHub ↗

License Apache-2.0Imported 8/10/2026Version v1

Skill contents

SKILL.md

Strategy: Result Analysis

Key Question: What do the results tell us?

Methodology

Three-layer analysis combining frequentist, resampling, and Bayesian approaches:

Statistical Testing — Bootstrap CI, Permutation tests, Bayesian ROPE judgment
Effect Size Calculation — Cohen's d, Cliff's delta, or domain-appropriate measure
Reproducibility Verification — Re-run with different seeds, compare distributions
Synthesis — Integrate findings into actionable conclusions

Execution Flow

[Collected results from experiment-running]
    → statistical-testing (bootstrap/permutation/Bayesian)
        → effect size calculation
            → reproducibility-verification (re-run, compare)
                → execution-synthesis (comprehensive report)
                    → OUTPUT: validated findings with confidence levels

Budget Gate

| Step | Max Budget | Output | |------|-----------|--------| | Statistical testing | 8% | Test results with p-values/CIs | | Reproducibility | 8% | Re-run comparison | | Synthesis | 4% | Final report |

Key Decisions

Test selection:
Known distribution → parametric (t-test, ANOVA)
Unknown/non-normal → bootstrap CI or permutation test
Need practical significance → Bayesian ROPE
Reproducibility threshold: Results must agree within 1 SE across re-runs
Effect size interpretation:
Small: d < 0.2 (may not be practically significant)
Medium: 0.2 ≤ d < 0.8 (likely meaningful)
Large: d ≥ 0.8 (strong effect)
ROPE (Region of Practical Equivalence): Define before testing, not after

Integration with Knowledge System

Results feed back into:

Wiki vault (claims with evidence)
Future experiment design (what worked, what didn't)
North star progress tracking

Available Tactics

Optional, no fixed order; the final leaf is always a sop.

| Tactic | When to use | | --- | --- | | result-validation-loop | Validate results through statistical testing, ROPE judgment, reproducibility re-runs, and final synthesis |

Available SOPs

Optional, no fixed order; the final leaf is always a sop.

| SOP | When to use | | --- | --- | | execution-synthesis | Synthesize complete execution report from all results, tests, and reproducibility data | | reproducibility-verification | Verify result reproducibility via re-runs with different seeds and ICC comparison | | statistical-testing | Execute statistical tests — bootstrap, permutation, Bayesian ROPE — on experiment results |

Illustrative Want to see what yogsoth-ai/result-analysis would put in front of your model? Preview the rendered body and check its trigger phrases — no LLM call, no signup.

Latest benchmark · publisher runVerified

9/3/2026, 15:56 UTC · 24 cases

Lift

+42%

75% vs 33%

Agent turns

0%

1.0 vs 1.0

Tokens

+3%

2,931.27 vs 2,833.59

fail → pass · 11pass → pass · 7fail → fail · 5pass → fail · 1

Per-case results (24)

Case	Status	Duration (ms)			Turns			Tokens			Tool calls
Case	Status	Without	With	Δ	Without	With	Δ	Without	With	Δ	Without	With	Δ
case-01 I've collected evaluation metrics from running two retrieval pipeline variants across ten random seed iterations. Could you analyze these experimental outputs to see if there is a real difference between the variants, verify if the findings hold across re-runs, and generate a comprehensive summary report with confidence estimates?	fail→pass	30,935	23,062	-25%	1	1	0%	5,568	4,036	-28%	0	0	—
case-02 Here are the test accuracy scores from our baseline model vs our fine-tuned candidate model across 5 distinct evaluation runs. Please perform a statistical analysis on these performance results, assess the magnitude of the improvement, verify consistency across runs, and synthesize the final findings into a structured report.	fail→fail	13,877	33,952	+145%	1	1	0%	2,717	6,752	+149%	0	0	—
case-03 We recently completed a batch of prompt optimization trials and gathered raw latency and accuracy numbers across multiple seeds. I need you to process these experimental findings through statistical validation, evaluate whether the performance gains are practically meaningful, verify seed reproducibility, and provide an actionable conclusion report.	fail→fail	36,747	27,487	-25%	1	1	0%	8,241	5,155	-37%	0	0	—
case-04 We evaluated our new LLM reranker against BM25 on a small set of 15 queries. The score delta distribution is heavily skewed with extreme outliers, so standard normality assumptions fail. A teammate suggested relying solely on a standard paired Student's t-test. How should we perform statistical testing on these metric deltas?	fail→pass	19,662	27,314	+39%	1	1	0%	2,595	4,607	+78%	0	0	—
case-05 We measured latency across 10,000 requests for two server configurations. The response times are strictly log-normal, and taking log transformations yields perfectly normally distributed data with equal variance. Should we use parametric statistical tests or non-parametric bootstrap resampling for comparing log-transformed mean latencies?	pass→pass	18,834	14,471	-23%	1	1	0%	2,212	3,103	+40%	0	0	—
case-06 Our data science team finished running a hypothesis test on dataset extraction speed, got a p-value of 0.03, and now wants to construct the Region of Practical Equivalence (ROPE) bounds to see if p=0.03 is practically meaningful. What is the correct protocol regarding when ROPE bounds must be defined?	pass→pass	18,316	22,248	+21%	1	1	0%	2,311	2,263	-2%	0	0	—
case-07 In our benchmark re-runs across three random seeds, the candidate model achieved metric means of 84.2, 85.1, and 84.8, with a standard error (SE) of 0.45 across runs. The lead engineer wants to use a 3% relative variance tolerance to pass reproducibility. What exact quantitative threshold should be used to verify reproducibility between re-runs?	fail→pass	47,237	27,110	-43%	1	1	0%	3,053	5,144	+68%	0	0	—
case-08 An A/B experiment comparing search ranking algorithms yields a Cohen's d of 0.15 with p = 0.001 due to a sample size of 500,000 records. The product owner wants to declare this a major breakthrough based on p-value alone. How should this effect size of d = 0.15 be interpreted for decision making?	pass→pass	16,163	13,343	-17%	1	1	0%	2,592	3,086	+19%	0	0	—
case-09 A model distillation trial yields a Cohen's d of 0.45 in inference throughput improvement. The team lead asks whether d = 0.45 represents a small, medium, or large effect according to standard categorization.	pass→fail	8,240	4,053	-51%	1	1	0%	1,315	1,385	+5%	0	0	—
case-14 We have raw collected experimental results from our multi-agent framework test suite. What is the prescribed linear execution order of analysis tactics from raw results to final validated findings?	fail→pass	17,666	7,406	-58%	1	1	0%	2,826	1,882	-33%	0	0	—
case-10 Replacing our baseline vector index with a GPU-accelerated HNSW index reduced query latency with a Cohen's d of 0.95. How should we categorize this magnitude of improvement?	pass→pass	10,297	10,809	+5%	1	1	0%	1,901	2,617	+38%	0	0	—
case-11 We are allocating execution budget across our analysis pipeline steps (Statistical Testing, Reproducibility Verification, and Synthesis). A team member suggests allocating 25% of the overall analysis budget to running initial statistical tests (bootstrap, permutation, ROPE). What is the maximum budget limit for the statistical testing step?	fail→pass	9,469	12,413	+31%	1	1	0%	689	996	+45%	0	0	—
case-12 When setting resource constraints for the result analysis phase, how much budget allocation is capped for the reproducibility verification re-run comparison step?	fail→pass	26,296	7,807	-70%	1	1	0%	1,010	965	-4%	0	0	—
case-13 Our final synthesis report step is taking significant computing and context budget. What is the maximum budget cap assigned to the final synthesis phase in the execution workflow?	fail→pass	7,624	1,708	-78%	1	1	0%	938	889	-5%	0	0	—
case-15 After validating experimental findings with statistical testing and reproducibility re-runs, how should these verified claims and evidence be integrated back into the team's long-term knowledge repository?	pass→pass	23,771	13,946	-41%	1	1	0%	2,706	2,684	-1%	0	0	—
case-16 We need an end-to-end strategy tactic that combines statistical testing, ROPE judgment, reproducibility re-runs across seeds, and final report synthesis into a single structured validation loop for our benchmark dataset. Which specific tactic covers this entire sequence?	fail→pass	17,798	6,480	-64%	1	1	0%	2,858	1,203	-58%	0	0	—
case-17 Our statistical testing, effect size calculations, and seed re-runs are complete. Which specific standard operating procedure (SOP) should be executed to pull together all p-values, CIs, ICC comparisons, and seed metrics into a comprehensive execution report?	fail→pass	9,576	33,244	+247%	1	1	0%	1,417	1,183	-17%	0	0	—
case-18 We have finished initial statistical testing on run 1 of our experiment. Which dedicated SOP handles verifying result reproducibility across different seed re-runs using ICC comparison?	pass→pass	12,334	19,494	+58%	1	1	0%	1,927	876	-55%	0	0	—
case-19 We need to run bootstrap resampling, permutation tests, and Bayesian ROPE on raw experiment output tables. Which SOP is explicitly dedicated to executing these specific statistical tests?	fail→pass	10,990	20,166	+83%	1	1	0%	1,542	971	-37%	0	0	—
case-20 We want to set up an automated hyperparameter optimization sweep using Optuna for our Transformer cross-encoder model. Can you define the search space bounds, learning rate decay schedules, and select the TPE sampler parameters for this training job?	fail→fail	27,565	27,206	-1%	1	1	0%	4,352	5,688	+31%	0	0	—
case-21 We need to set up OpenTelemetry collectors and Kafka topics to stream real-time GPU memory usage and CPU utilization metrics from our Kubernetes training cluster. How should we configure the collector YAML and Kafka consumer consumer-groups?	fail→fail	30,090	27,263	-9%	1	1	0%	4,582	4,879	+6%	0	0	—
case-22 Can you author a system prompt and zero-shot chain-of-thought template for extracting medical entities from patient clinical notes, including few-shot edge case examples?	fail→fail	25,661	35,696	+39%	1	1	0%	5,108	6,901	+35%	0	0	—
case-23 Our evaluation plan currently only computes p-values from standard t-tests and outputs a mean difference table. What three computational methodologies must be combined in the initial statistical testing layer to meet full result validation standards?	fail→pass	25,504,516	4,391	-100%	1	1	0%	1,747	1,448	-17%	0	0	—
case-24 We evaluated LLM response safety scores on 30 test prompts. The score distributions are discrete, non-normal, and bounded between 1 and 5. A colleague recommends fitting a parametric ANOVA model. What test approach should be selected according to key decision rules for non-normal distributions?	pass→pass	13,869	8,267	-40%	1	1	0%	2,195	1,884	-14%	0	0	—

These cases were also screened by the skill’s intent / safety pass.See the Trust review ↗Reproduce this benchmark in Colab ↗

How this skill was measured

DecimalAI ran this skill against gemini-3.6-flash twice over the same eval suite — once with the skill loaded and once without — and compared the two runs case by case. 24 cases were attempted, and 22 counted toward the lift figure. The other 2 produced results that are not comparable between the two arms, so they are excluded from the headline rather than averaged into it. The headline lift of +42 percentage points is the difference between those two pass rates over the 22 comparable cases. 1 case got worse with the skill loaded, and it is included in that figure.

Model measured: gemini-3.6-flash
Graded by: gemini-3.6-flash
Grading method: judged
Verification: verified
Pass rate without the skill: 33.3%
Pass rate with the skill: 75%
Cases attempted: 24
Cases counted toward lift: 22
Cases that regressed: 1
Skill version measured: v1
Run completed: 9/3/2026, 15:56 UTC

Measured example — same prompt, with vs. without the skill · from the benchmark run · case case-01

Prompt

I've collected evaluation metrics from running two retrieval pipeline variants across ten random seed iterations. Could you analyze these experimental outputs to see if there is a real difference between the variants, verify if the findings hold across re-runs, and generate a comprehensive summary report with confidence estimates?

Without the skill loaded, the model failed this case. With it loaded, the same prompt on the same model passed. This is one improved case from the latest verified run; every case, including any that regressed, is in the table above.

Show both answers

SkillScore blends 4 signals1 of 4 complete

BenchmarkPass rate WITH skill on a verified run75

Live evalNeeds ≥10 evaluated traces in 30 days

AI ratingNeeds ≥10 rated traces

AdoptionNeeds ≥10 serves across ≥2 orgs in 30 days

Efficiency · separate axis (at equal correctness)+0% turns+4% tokens

Shown for transparency; not credited unless a verified run shows no correctness regression.

Backed by 1 of 4 signals — each added signal makes the score more robust.

Verified on gemini-3.6-flash · re-tested 18 days ago

Other measured skills in the registry, with their headline benchmark lift.