Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

Get Started Free →

Tactic: Reproducibility Protocol · yogsoth-ai | DecimalAI

yogsoth-ai/reproducibility-protocol

SkillScore77Maintained

Ensure experiment reproducibility through systematic environment and seed control

researchby yogsoth-ai

Links this skill to your agents and keeps it current with the author’s updates.

Automatic

AI Engineer pack

1 · One-time — install the CLI (docs ↗)

pip install decimalai

2 · Get the skill — writes .claude/skills/yogsoth-ai-reproducibility-protocol/SKILL.md

decimalai skills pull yogsoth-ai-reproducibility-protocol --out .claude/skills/

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/yogsoth-ai-reproducibility-protocol/SKILL.md https://app.decimal.ai/s/yogsoth-ai-reproducibility-protocol/SKILL.md

Raw file: /s/yogsoth-ai-reproducibility-protocol/SKILL.md ↗ · machine-readable summary at /s/yogsoth-ai-reproducibility-protocol.json ↗

In productionGlobal

how this skill behaves on real traffic · last 30 days

No production data for this skill yet — usage numbers appear here once agents start running it.

Anonymized aggregate across every org using this skill — proof from real production use, not a demo.

Does this skill help? · Benchmark: with vs. withoutsign in to run this on your agents

22 test cases from the publisher’s eval.yaml · each run twice on gemini-3.6-flash — with the skill injected, then without · last run 18d ago

—+18% tokens0% turns77% of whole cases pass with skill (17/22)

Test case	Without → With	Effect	Δ tokens	Δ turns
case-06	✗→✓	▲ Improved	42%	0%
case-09	✗→✓	▲ Improved	41%	0%
case-10	✗→✓	▲ Improved	22%	0%
case-12	✗→✓	▲ Improved	-24%	0%
case-13	✗→✓	▲ Improved	-34%	0%

Showing 5 of 22 cases

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v4 — 8/10/2026

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

yogsoth-ai / de-anthropocentric-research-engine

skills/reproducibility-protocol/SKILL.md

View on GitHub ↗

License Apache-2.0Imported 8/10/2026Version v1

Skill contents

SKILL.md

Tactic: Reproducibility Protocol

Orchestration Pattern

Assess Reproducibility Requirements → Determine level needed (exact, statistical, conceptual)
seed-protocol-design → Design random seed strategy for all stochastic components
environment-specification → Lock all software, hardware, and configuration dependencies
Define Verification Plan → How to confirm reproducibility (re-run subset, cross-machine test)
Document Non-Determinism → Identify and document unavoidable sources of variance

Decision Criteria

| Reproducibility Level | Requirement | When to Use | |----------------------|-------------|-------------| | Exact (bit-for-bit) | Same hardware + deterministic ops | Debugging, verification | | Statistical (within CI) | Same distribution of results | Standard research | | Conceptual (same conclusion) | Same qualitative findings | Cross-platform validation |

Quality Checks

Are all random seeds documented and controllable?
Is the full software environment captured (versions, dependencies)?
Are hardware-specific non-determinisms identified (GPU atomics, cuDNN)?
Is there a verification protocol (re-run N times, check variance)?
Are data preprocessing steps deterministic or seeded?
Is the order of data loading controlled?
Are model initialization seeds separate from data shuffling seeds?
Is there a plan for cross-machine reproducibility testing?

Available SOPs

Optional, no fixed order; the final leaf is always a sop.

| SOP | When to use | | --- | --- | | environment-specification | SOP: define complete experiment environment specification | | seed-protocol-design | SOP: design random seed strategy for reproducibility |

Illustrative Want to see what yogsoth-ai/reproducibility-protocol would put in front of your model? Preview the rendered body and check its trigger phrases — no LLM call, no signup.

Latest benchmark · publisher runVerified

9/3/2026, 08:39 UTC · 22 cases

Measured on 21 comparable cases — below the 22 needed to publish a lift, so this result is shown here but not on the registry card or leaderboard.

Lift

+36%

77% vs 41%

Agent turns

0%

1.0 vs 1.0

Tokens

+18%

2,921.67 vs 2,479.62

fail → pass · 8pass → pass · 9fail → fail · 5

Per-case results (22)

Case	Status	Duration (ms)			Turns			Tokens			Tool calls
Case	Status	Without	With	Δ	Without	With	Δ	Without	With	Δ	Without	With	Δ
case-01 We are preparing a machine learning research paper submission, but our model training results fluctuate slightly between different GPU nodes. Can you help us draft a complete reproducibility plan that covers setting up random seed controls, capturing environment specs, and defining how to verify our results?	fail→fail	26,666	29,864	+12%	1	1	0%	3,533	4,907	+39%	0	0	—
case-02 Our team needs to establish a clear reproducibility policy for an upcoming audit of our predictive analytics pipeline. Please outline a step-by-step strategy for managing stochastic components, locking dependencies, and accounting for potential sources of hardware variance during re-runs.	fail→fail	23,832	30,563	+28%	1	1	0%	3,352	5,116	+53%	0	0	—
case-03 I'm encountering inconsistent behavior while trying to debug a complex simulation experiment across multiple developer machines. Could you design a comprehensive protocol for environment pinning, seed distribution, and verification testing so we can consistently replicate our benchmark runs?	fail→fail	26,171	32,096	+23%	1	1	0%	4,511	5,287	+17%	0	0	—
case-04 We want to configure an automated hyperparameter search using Bayesian optimization for an XGBoost model. How should we set up the search space, acquisition function, and stopping criteria to maximize validation ROC-AUC?	pass→pass	28,807	31,630	+10%	1	1	0%	3,939	5,373	+36%	0	0	—
case-05 Our Git commit history on the main development branch is messy with dozens of intermediate fixup commit messages. What is the standard procedure for squashing intermediate commits and rewriting branch history before merging?	pass→pass	13,055	14,116	+8%	1	1	0%	2,344	2,762	+18%	0	0	—
case-06 Our GitHub Actions build pipeline takes 25 minutes because it installs heavy Rust compiler tools and Python wheels on every trigger. How should we configure pipeline dependency caching to reduce build times?	fail→pass	19,557	19,941	+2%	1	1	0%	2,737	3,882	+42%	0	0	—
case-07 When training deep neural networks, engineers often set a single global random seed at the top of the main script and assume that covers all randomness. How should random seed management be structured across different stochastic components?	pass→pass	18,221	22,899	+26%	1	1	0%	3,231	3,421	+6%	0	0	—
case-08 We set framework and numerical random seeds to 42, but running model training twice on Nvidia A100 GPUs produces slightly different loss curves. What specific low-level library operations and hardware features cause this non-determinism?	pass→pass	28,751	23,274	-19%	1	1	0%	3,565	3,371	-5%	0	0	—
case-09 A financial regulatory compliance team requires proof that our Monte Carlo risk model produces identical results during an audit. Should we aim for statistical tolerance matching or exact matching, and what conditions are required?	fail→pass	19,926	23,588	+18%	1	1	0%	2,932	4,126	+41%	0	0	—
case-10 We are migrating an image classification benchmark across different hardware architectures and backend frameworks. Exact bitwise numerical matching is impossible due to framework floating-point differences. Which level of reproducibility should be targeted and what is its goal?	fail→pass	14,885	17,580	+18%	1	1	0%	2,124	2,585	+22%	0	0	—
case-11 Our distributed PyTorch DataLoader uses multiple worker processes with asynchronous prefetching. Even with fixed random seeds, batch contents arrive in variable order across epochs. What quality check must be applied to data loading?	fail→fail	16,519	14,299	-13%	1	1	0%	2,143	2,355	+10%	0	0	—
case-12 After determining our project requires statistical reproducibility across CI runs, what is the immediate next step in the reproducibility orchestration workflow?	fail→pass	8,514	4,925	-42%	1	1	0%	1,193	908	-24%	0	0	—
case-13 Once a comprehensive random seed strategy is designed for a machine learning experiment, what orchestration step must be performed prior to defining verification plans?	fail→pass	9,982	4,541	-55%	1	1	0%	1,302	856	-34%	0	0	—
case-14 We have pinned CUDA versions, container dependencies, and set all random seeds, but parallel reduction ops on GPUs still introduce tiny floating-point variance. What is the final step in the orchestration pattern for handling this variance?	fail→pass	28,549	7,497	-74%	1	1	0%	1,439	1,159	-19%	0	0	—
case-15 A team lead claims that writing unit tests for individual functions is sufficient to prove an end-to-end ML training pipeline is reproducible. What verification protocol quality check should be required instead?	fail→pass	18,724	9,832	-47%	1	1	0%	2,090	1,841	-12%	0	0	—
case-16 Our automated CI benchmark runs model evaluation on every pull request. We want to catch regressions without failing builds due to minor floating-point jitter. Which reproducibility level applies and what does it require?	fail→fail	22,489	19,684	-12%	1	1	0%	2,313	2,329	+1%	0	0	—
case-17 A computer vision pipeline performs random image cropping, color jitter, and rotation during training data preparation. What specific check must be performed on data preprocessing to guarantee reproducibility?	pass→pass	11,985	36,955	+208%	1	1	0%	1,944	2,480	+28%	0	0	—
case-18 An engineer saved only an unpinned package list file without exact versions or configuration flags. What specific quality check requirement does this fail?	pass→pass	25,122	10,689	-57%	1	1	0%	1,486	1,254	-16%	0	0	—
case-19 After locking all container images, environment dependencies, and random seed strategies, what step should be executed before documenting non-deterministic residual variance?	fail→pass	10,138	15,135	+49%	1	1	0%	1,110	911	-18%	0	0	—
case-20 Our deep learning model trains on cloud GPU instances but researchers run initial validation on local workstation GPUs. What specific quality check ensures results match across these environments?	pass→pass	24,928	21,257	-15%	1	1	0%	2,016	2,040	+1%	0	0	—
case-21 A research team is drafting a paper on a new reinforcement learning algorithm for a standard benchmark suite. Should they mandate bit-for-bit identity or statistical consistency for standard research submission?	pass→pass	17,374	29,376	+69%	1	1	0%	1,980	2,443	+23%	0	0	—
case-22 A developer wants to achieve bit-for-bit exact reproducibility across both AMD GPUs and NVIDIA GPUs using deterministic flags. Is exact bit-for-bit identity achievable across different GPU architectures?	pass→pass	22,920	16,379	-29%	1	1	0%	2,227	3,108	+40%	0	0	—

These cases were also screened by the skill’s intent / safety pass.See the Trust review ↗Reproduce this benchmark in Colab ↗

How this skill was measured

DecimalAI ran this skill against gemini-3.6-flash twice over the same eval suite — once with the skill loaded and once without — and compared the two runs case by case. 22 cases were attempted, and 21 counted toward the lift figure. The other 1 produced results that are not comparable between the two arms, so they are excluded from the headline rather than averaged into it. The headline lift of +36 percentage points is the difference between those two pass rates over the 21 comparable cases.

Model measured: gemini-3.6-flash
Graded by: gemini-3.6-flash
Grading method: judged
Verification: verified
Pass rate without the skill: 40.9%
Pass rate with the skill: 77.3%
Cases attempted: 22
Cases counted toward lift: 21
Cases that regressed: 0
Skill version measured: v1
Run completed: 9/3/2026, 08:39 UTC

Measured example — same prompt, with vs. without the skill · from the benchmark run · case case-06

Prompt

Our GitHub Actions build pipeline takes 25 minutes because it installs heavy Rust compiler tools and Python wheels on every trigger. How should we configure pipeline dependency caching to reduce build times?

Without the skill loaded, the model failed this case. With it loaded, the same prompt on the same model passed. This is one improved case from the latest verified run; every case, including any that regressed, is in the table above.

Show both answers

SkillScore blends 4 signals1 of 4 complete

BenchmarkPass rate WITH skill on a verified run77

Live evalNeeds ≥10 evaluated traces in 30 days

AI ratingNeeds ≥10 rated traces

AdoptionNeeds ≥10 serves across ≥2 orgs in 30 days

Efficiency · separate axis (at equal correctness)+0% turns+16% tokens

Shown for transparency; not credited unless a verified run shows no correctness regression.

Backed by 1 of 4 signals — each added signal makes the score more robust.

Verified on gemini-3.6-flash · re-tested 18 days ago

Other measured skills in the registry, with their headline benchmark lift.