Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

Get Started Free →

SOP: Experiment Config Generation · yogsoth-ai | DecimalAI

yogsoth-ai/experiment-config-generation

SkillScore17Maintained

SOP: generate executable experiment configuration files

AI/MLby yogsoth-ai

Links this skill to your agents and keeps it current with the author’s updates.

Automatic

AI Engineer pack

1 · One-time — install the CLI (docs ↗)

pip install decimalai

2 · Get the skill — writes .claude/skills/yogsoth-ai-experiment-config-generation/SKILL.md

decimalai skills pull yogsoth-ai-experiment-config-generation --out .claude/skills/

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/yogsoth-ai-experiment-config-generation/SKILL.md https://app.decimal.ai/s/yogsoth-ai-experiment-config-generation/SKILL.md

Raw file: /s/yogsoth-ai-experiment-config-generation/SKILL.md ↗ · machine-readable summary at /s/yogsoth-ai-experiment-config-generation.json ↗

In productionGlobal

how this skill behaves on real traffic · last 30 days

No production data for this skill yet — usage numbers appear here once agents start running it.

Anonymized aggregate across every org using this skill — proof from real production use, not a demo.

Does this skill help? · Benchmark: with vs. withoutsign in to run this on your agents

23 test cases from the publisher’s eval.yaml · each run twice on gemini-3.6-flash — with the skill injected, then without · last run 7d ago

—-5% tokens0% turns17% of whole cases pass with skill (4/23)

Test case	Without → With	Effect	Δ tokens	Δ turns
case-03	✗→✓	▲ Improved	-74%	0%
case-20	✓→✓	＝ Same ✓	32%	0%
case-21	✓→✓	＝ Same ✓	40%	0%
case-22	✓→✓	＝ Same ✓	-45%	0%
case-01	✗→✗	＝ Same ✗	2%	0%

Showing 5 of 23 cases

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v4 — 8/3/2026

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

yogsoth-ai / de-anthropocentric-research-engine

skills/experiment-config-generation/SKILL.md

View on GitHub ↗

License Apache-2.0Imported 8/3/2026Version v1

Skill contents

SKILL.md

SOP: Experiment Config Generation

Synthesize all experiment design elements (design matrix, environment, seeds, metrics) into directly executable configuration files, enabling seamless transition from design to execution.

Execution

Subagent — spawned via subagent-spawning/spawn-agent skill.

Why Subagent

Config generation requires integrating outputs from multiple upstream SOPs, transforming abstract experiment designs into concrete executable specifications — involves format conversion and consistency validation.

Available SOPs

Optional, no fixed order; the final leaf is always a sop.

| SOP | When to use | | --- | --- | | spawn-agent | Spawn a customized CC subagent with full MCP tool access. Used by SOPs that declare execution: subagent. |

Want to see if yogsoth-ai/experiment-config-generation would activate on your phrasing? Try it before you install — no LLM call, no signup.

Latest benchmark · publisher runVerified

8/23/2026, 01:59 UTC · 23 cases

Measured on 21 comparable cases — below the 22 needed to publish a lift, so this result is shown here but not on the registry card or leaderboard.

Lift

+4%

17% vs 13%

Agent turns

0%

1.0 vs 1.0

Tokens

-5%

2,676.14 vs 2,822.62

fail → pass · 1pass → pass · 3fail → fail · 19

Per-case results (23)

Case	Status	Duration (ms)			Turns			Tokens			Tool calls
Case	Status	Without	With	Δ	Without	With	Δ	Without	With	Δ	Without	With	Δ
case-01 I have finalized the design matrix, environmental variables, random seeds, and evaluation metrics for our multi-agent RL benchmark. Could you synthesize these abstract parameters into fully executable configuration files that our runner can parse directly?	fail→fail	20,681	27,157	+31%	1	1	0%	3,746	3,809	+2%	0	0	—
case-02 Please convert our experiment design specifications—including seed selections, metric schemas, environment details, and variable matrices—into ready-to-use run configs. Make sure the output format is fully validated and ready for execution.	fail→fail	12,150	4,135	-66%	1	1	0%	2,274	768	-66%	0	0	—
case-03 We need to translate the proposed hyperparameter search matrix, evaluation metrics, seed lists, and runtime environment settings into concrete, runnable config files for our training pipeline.	fail→pass	11,803	2,951	-75%	1	1	0%	2,304	589	-74%	0	0	—
case-12 Our PPO continuous control task design specifies clip ranges [0.1, 0.2], learning rates, seeds 1 to 5, and Mujoco environment configs. Synthesize these design elements into executable run files.	fail→fail	50,753	28,272	-44%	1	1	0%	1,241	572	-54%	0	0	—
case-04 Here are our computer vision ablation study parameters: 5 random seeds, resnet50 and vit-b backbones, batch sizes 32 and 64, and mAP@50 evaluation metrics. Please assemble these into ready-to-execute YAML configs for our training cluster.	fail→fail	16,789	14,201	-15%	1	1	0%	3,541	3,132	-12%	0	0	—
case-05 We have defined our LLM fine-tuning experiment: LoRA rank [8, 16], learning rate [1e-4, 5e-5], environment set to CUDA_VISIBLE_DEVICES=0,1, and validation metric eval_loss. Please write out the executable JSON experiment configuration files right now.	fail→fail	12,436	11,739	-6%	1	1	0%	2,856	2,954	+3%	0	0	—
case-06 Our GNN graph classification study has 3 dataset splits, seeds 42, 123, 999, and ROC-AUC evaluation metrics. Please synthesize these abstract design specifications into executable config files.	fail→fail	29,195	13,579	-53%	1	1	0%	2,774	2,847	+3%	0	0	—
case-07 We have completed our speech recognition experiment design with target metrics (WER, CER), hardware environment settings, and noise level parameters. Please convert this into executable config files.	fail→fail	14,138	4,704	-67%	1	1	0%	2,638	652	-75%	0	0	—
case-08 Our recommendation system experiment specifies NDCG@10 metrics, implicit feedback matrices, and PyTorch environment settings. Synthesize these into executable configurations.	fail→fail	19,718	17,239	-13%	1	1	0%	4,071	3,558	-13%	0	0	—
case-09 We have selected sampler types [DDIM, Euler], step counts [20, 50], and FID score metrics for image generation testing. Generate the executable runtime config files.	fail→fail	28,756	12,324	-57%	1	1	0%	1,094	2,566	+135%	0	0	—
case-10 Our time-series forecasting design includes ARIMA vs Informer baselines, lookback horizons [24, 48, 96], MSE/MAE metrics, and seed list [101, 202]. Convert these design specs into executable pipeline configs.	fail→fail	15,511	14,749	-5%	1	1	0%	3,092	3,436	+11%	0	0	—
case-11 We have structured a prompt engineering comparison matrix across 4 model endpoints, 3 seed variations, and accuracy/latency metrics. Please transform this into executable configuration files.	fail→fail	17,948	14,679	-18%	1	1	0%	3,420	3,168	-7%	0	0	—
case-13 We have finalized our BERT sequence labeling experiment matrix: max length [128, 256], dropout [0.1, 0.3], macro-F1 evaluation, and seed array. Generate all executable config files.	fail→fail	19,353	15,165	-22%	1	1	0%	4,743	3,455	-27%	0	0	—
case-14 Our multi-task learning study defines loss weighting strategies, seed values, and multi-task evaluation metrics. Convert these abstract parameters into executable configs.	fail→fail	21,611	5,912	-73%	1	1	0%	4,331	617	-86%	0	0	—
case-15 We have designed a backtesting grid for stock trading algorithms including historical date ranges, transaction fee parameters, Sharpe ratio metrics, and seed parameters. Synthesize these into executable execution configs.	fail→fail	39,019	24,095	-38%	1	1	0%	3,280	5,013	+53%	0	0	—
case-16 Our federated learning experiment design sets client selection ratios [0.1, 0.5], local epoch counts, communication rounds, and accuracy metrics. Generate executable configuration files for each run.	fail→fail	14,947	18,609	+24%	1	1	0%	3,302	4,159	+26%	0	0	—
case-17 We have chosen UNet vs VNet architectures, Dice score evaluation, data augmentation flags, and seed seeds [7, 14, 21] for our medical image segmentation study. Please turn these into executable run configs.	fail→fail	15,487	20,109	+30%	1	1	0%	2,998	4,642	+55%	0	0	—
case-18 Our speech separation experiment matrix specifies SDR/SIR evaluation metrics, STFT window sizes, dataset splits, and random seeds. Synthesize these abstract specs into directly executable config files.	fail→fail	14,880	23,370	+57%	1	1	0%	2,956	5,261	+78%	0	0	—
case-19 Our cross-lingual NLI experiment design includes 5 languages, batch size 16, evaluation metrics (accuracy per language), and seed list. Please output the executable run config files.	fail→fail	13,294	14,253	+7%	1	1	0%	2,661	3,155	+19%	0	0	—
case-20 We want to execute a hyperparameter sweep over multiple learning rate parameters directly from the terminal without creating a subagent or generating new config files. Which command-line flag in Hydra enables multirun mode for parameter sweeps?	pass→pass	2,223	2,424	+9%	1	1	0%	376	497	+32%	0	0	—
case-21 We are writing a Python script to tune experiment hyperparameters using Optuna's Tree-structured Parzen Estimator algorithm. What class name should be imported from `optuna.samplers` to instantiate this sampler?	pass→pass	1,983	1,940	-2%	1	1	0%	365	512	+40%	0	0	—
case-22 We want to automatically log model hyperparameters, metrics, and artifacts during PyTorch model training runs using MLflow. What Python function call in MLflow enables automatic logging for PyTorch Lightning models?	pass→pass	5,174	3,057	-41%	1	1	0%	918	503	-45%	0	0	—
case-23 Our quadruped robot simulation experiment has specified physics engine parameters, friction coefficients, reward weights, and 10 random seeds. Convert these specs into executable configuration files.	fail→fail	13,888	39,593	+185%	1	1	0%	2,629	3,472	+32%	0	0	—

These cases were also screened by the skill’s intent / safety pass.See the Trust review ↗Reproduce this benchmark in Colab ↗

How this skill was measured

DecimalAI ran this skill against gemini-3.6-flash twice over the same eval suite — once with the skill loaded and once without — and compared the two runs case by case. 23 cases were attempted, and 21 counted toward the lift figure. The other 2 produced results that are not comparable between the two arms, so they are excluded from the headline rather than averaged into it. The headline lift of +4 percentage points is the difference between those two pass rates over the 21 comparable cases.

Model measured: gemini-3.6-flash
Graded by: gemini-3.6-flash
Grading method: judged
Verification: verified
Pass rate without the skill: 13%
Pass rate with the skill: 17.4%
Cases attempted: 23
Cases counted toward lift: 21
Cases that regressed: 0
Skill version measured: v1
Run completed: 8/23/2026, 01:59 UTC

Measured example — same prompt, with vs. without the skill · from the benchmark run · case case-03

Prompt

We need to translate the proposed hyperparameter search matrix, evaluation metrics, seed lists, and runtime environment settings into concrete, runnable config files for our training pipeline.

Without the skill loaded, the model failed this case. With it loaded, the same prompt on the same model passed. This is one improved case from the latest verified run; every case, including any that regressed, is in the table above.

Show both answers

SkillScore blends 4 signals1 of 4 complete

BenchmarkPass rate WITH skill on a verified run17

Live evalNeeds ≥20 evaluated traces in 30 days

AI ratingNeeds ≥10 rated traces

AdoptionNeeds ≥10 serves across ≥2 orgs in 30 days

Efficiency · separate axis (at equal correctness)+0% turns-9% tokens

Shown for transparency; not credited unless a verified run shows no correctness regression.

Backed by 1 of 4 signals — each added signal makes the score more robust.

Verified on gemini-3.6-flash · re-tested 6 days ago

Other measured skills in the registry, with their headline benchmark lift.