Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

Get Started Free →

Strategy: Comparison Design · yogsoth-ai | DecimalAI

ai mlby yogsoth-ai

Strategy: Comparison Design

yogsoth-ai/comparison-design

Source yogsoth-ai/de-anthropocentric-research-engine ↗License Apache-2.0

Design fair comparison experiments against baselines and competing methods

SkillScore84Maintained

Automatic

Research pack

Last verified 8/22/2026

What it changes

Passed 19 of 23 test cases with the skill, and 7 of 23 without it.

Without skill32%

With skill84%

Checks it passed that the model alone missed

Specifies a minimum threshold of 200 runs.
Specifies evaluating on at least 5 datasets.
Specifies a minimum run volume of 30-45 runs.

and 17 more in the full report

13 fixed8 passed either way4 still failing0 broke

Pass-rate lift: +52 pp
Tokens: +10%
Agent turns: 0%
Response time: -17%

Model gemini-3.6-flashMeasured 8/22/2026v121 of 25 cases passed with skill · 23 comparable

Tokens, turns and time describe resource use, not a dollar price.

Use this skill

Save it into your project's skills folder. Claude Code picks it up in its next session.

curl -fsSL --create-dirs -o .claude/skills/yogsoth-ai-comparison-design/SKILL.md https://app.decimal.ai/s/yogsoth-ai-comparison-design/SKILL.md

Want it pinned and updatable? pip install "decimalai>=0.15", then decimalai skills add yogsoth-ai-comparison-design, and start Claude Code with claude --plugin-dir "$(git rev-parse --show-toplevel)/.agents"

1 · One-time — install the CLI (docs ↗)

pip install "decimalai>=0.15"

2 · Get the skill — writes .agents/skills/yogsoth-ai-comparison-design/ and pins it in .decimal/skills.lock

decimalai skills add yogsoth-ai-comparison-design

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/yogsoth-ai-comparison-design/SKILL.md https://app.decimal.ai/s/yogsoth-ai-comparison-design/SKILL.md

Raw file: /s/yogsoth-ai-comparison-design/SKILL.md ↗ · machine-readable summary at /s/yogsoth-ai-comparison-design.json ↗

Safety checks: 3 of 3 passed

Code scan CleanIntent PassedContent Clean

See it in action

One of 11 test cases the skill fixed. Same model, same prompt, with and without the skill.

1 of 11

Prompt

If we compare 1 SOTA baseline and 1 simple baseline against our method on 1 dataset using 3 random seeds, how many total runs are required under the Minimal scope gate?

What was checked, without and with the skill
Checked	Without skill	With skill
Specifies a minimum requirement of 6 total experimental runs.	Missed	Passed

case-23 · gemini-3.6-flash · 8/22/2026 · answers are capped and redacted

What's inside

The instructions your model reads when the skill loads.

SKILL.md55 linessingle file

Strategy: Comparison Design

Question: How much better is our method than the baseline?

Methodology

Fair Comparison Protocol (Bouthillier 2021): Control all confounds, same compute budget, same tuning effort.
Multi-Baseline Comparison: Compare against multiple baselines (SOTA, simple, ablated).
Multi-Dataset Evaluation: Test across diverse datasets to avoid dataset-specific overfitting.
Bayesian Comparison (Benavoli 2017): Posterior probability of superiority, not just p-values.
Bootstrap/Permutation Tests: Non-parametric significance without distributional assumptions.

Execution Flow

baseline-selection → Select appropriate baselines (SOTA, simple, oracle)
metric-specification → Define primary metric and secondary metrics
sample-size-estimation → Power analysis for detecting meaningful differences
seed-protocol-design → Ensure fair random initialization across methods
environment-specification → Lock environment to prevent confounds
reproducibility-protocol (tactic) → Ensure all results are reproducible
statistical-method-selection (tactic) → Choose Bayesian or frequentist comparison

Budget Gate

Comparison Scope	Baselines	Datasets	Seeds	Min Runs
Minimal	1 SOTA + 1 simple	1	3	6
Standard	2-3 baselines	2-3	5	30-45
Comprehensive	4+ baselines	3-5	5-10	100+
Publication-ready	All relevant	5+	10+	200+

Available Tactics

Optional, no fixed order; the final leaf is always a sop.

Tactic	When to use
reproducibility-protocol	Ensure experiment reproducibility through systematic environment and seed control

Track record

Benchmark

19 / 23

test cases passed, vs 7 / 23 without

Adoption

Users: 0
Times used: 0

Safety and source details

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v4 — 8/3/2026

View 15 scanner checks

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

yogsoth-ai / de-anthropocentric-research-engine

skills/comparison-design/SKILL.md

View on GitHub ↗

License Apache-2.0Imported 8/3/2026Version v1

More skills from the same source and nearby in the registry, with their headline benchmark lift.

Ready to supercharge your AI agents?

See it in action

What's inside

Strategy: Comparison Design

Methodology

Execution Flow

Budget Gate

Available Tactics

Track record

Benchmark

Adoption

SkillSafety

Source

Related skills