Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

Get Started Free →

Strategy: Factor-Level Design · yogsoth-ai | DecimalAI

yogsoth-ai/experiment-execution-factor-level-design

SkillScore91Maintained

Design factorial experiments to test how specific factors affect outcomes

scienceby yogsoth-ai

Links this skill to your agents and keeps it current with the author’s updates.

Automatic

Research pack

1 · One-time — install the CLI (docs ↗)

pip install decimalai

2 · Get the skill — writes .claude/skills/yogsoth-ai-experiment-execution-factor-level-design/SKILL.md

decimalai skills pull yogsoth-ai-experiment-execution-factor-level-design --out .claude/skills/

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/yogsoth-ai-experiment-execution-factor-level-design/SKILL.md https://app.decimal.ai/s/yogsoth-ai-experiment-execution-factor-level-design/SKILL.md

Raw file: /s/yogsoth-ai-experiment-execution-factor-level-design/SKILL.md ↗ · machine-readable summary at /s/yogsoth-ai-experiment-execution-factor-level-design.json ↗

In productionGlobal

how this skill behaves on real traffic · last 30 days

No production data for this skill yet — usage numbers appear here once agents start running it.

Anonymized aggregate across every org using this skill — proof from real production use, not a demo.

Does this skill help? · Benchmark: with vs. withoutsign in to run this on your agents

22 test cases from the publisher’s eval.yaml · each run twice on gemini-3.6-flash — with the skill injected, then without · last run 7d ago

Yes — +14% lift+17% tokens0% turns91% of whole cases pass with skill (20/22)

Test case	Without → With	Effect	Δ tokens	Δ turns
case-01	✗→✓	▲ Improved	23%	0%
case-06	✗→✓	▲ Improved	17%	0%
case-09	✗→✓	▲ Improved	-1%	0%
case-03	✓→✓	＝ Same ✓	15%	0%
case-04	✓→✓	＝ Same ✓	18%	0%

Showing 5 of 22 cases

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v4 — 8/3/2026

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

yogsoth-ai / de-anthropocentric-research-engine

skills/experiment-execution-factor-level-design/SKILL.md

View on GitHub ↗

License Apache-2.0Imported 8/3/2026Version v1

Skill contents

SKILL.md

Strategy: Factor-Level Design

Question: Which factors to test at what levels in combination?

Methodology

Full Factorial (Fisher): All combinations of all factor levels. Gold standard but exponential cost.
Fractional Factorial: Systematic subset using defining relations. Sacrifices high-order interactions.
Plackett-Burman: Screening design for many factors. Identifies main effects only.
Response Surface Methodology (RSM): Central Composite / Box-Behnken for optimization after screening.
Taguchi Orthogonal Arrays: Robust design minimizing variance from noise factors.

Execution Flow

factor-identification → Identify all independent, dependent, and control variables
level-specification → Determine discrete levels for each factor (2-5 levels typical)
budget-constrained-design (tactic) → Select design type given budget constraints
design-matrix-construction → Build the actual design matrix
metric-specification → Define primary/secondary metrics and significance thresholds
sample-size-estimation → Power analysis to determine runs per cell

Budget Gate

| Design Type | Factors | Runs (k factors, 2 levels) | When to Use | |-------------|---------|---------------------------|-------------| | Full Factorial | 2-4 | 2^k | Budget allows, need all interactions | | Fractional (Res V) | 4-6 | 2^(k-1) | Need 2-factor interactions | | Fractional (Res III) | 5-8 | 2^(k-p) | Screening, main effects only | | Plackett-Burman | 8-15 | k+1 (nearest multiple of 4) | Many factors, screening phase | | Taguchi L9/L18 | 4-8 | 9 or 18 | Robust design with noise factors | | RSM (CCD) | 2-5 | 2^k + 2k + center | Optimization after screening |

Available Tactics

Optional, no fixed order; the final leaf is always a sop.

| Tactic | When to use | | --- | --- | | budget-constrained-design | Optimize experiment design under compute and time budget constraints | | statistical-method-selection | Select appropriate statistical methods for experiment analysis |

Available SOPs

Optional, no fixed order; the final leaf is always a sop.

| SOP | When to use | | --- | --- | | design-matrix-construction | Build the experiment design matrix with proper orthogonality and balance | | experiment-config-generation | SOP: generate executable experiment configuration files | | factor-identification | Identify independent, dependent, and control variables for an experiment | | level-specification | Determine appropriate levels for each experimental factor | | metric-specification | Define experiment metrics and significance standards | | sample-size-estimation | SOP: power analysis and required experiment count estimation |

Want to see if yogsoth-ai/experiment-execution-factor-level-design would activate on your phrasing? Try it before you install — no LLM call, no signup.

Latest benchmark · publisher runVerified

8/22/2026, 21:32 UTC · 22 cases

Lift

+14%

91% vs 77%

Agent turns

0%

1.0 vs 1.0

Tokens

+17%

2,595.64 vs 2,224.55

fail → pass · 3pass → pass · 17fail → fail · 2

Per-case results (22)

Case	Status	Duration (ms)			Turns			Tokens			Tool calls
Case	Status	Without	With	Δ	Without	With	Δ	Without	With	Δ	Without	With	Δ
case-01 I am preparing an AI fine-tuning experiment where I need to evaluate learning rate, batch size, dropout rate, weight decay, and sequence length simultaneously. Because GPU time is limited, we cannot test every single combination. Can you guide me through identifying the variables, picking a suitable experiment design method given our budget, outlining the level specifications, and detailing how to estimate sample sizes and metrics?	fail→pass	19,950	16,681	-16%	1	1	0%	3,238	3,983	+23%	0	0	—
case-02 We are setting up a benchmark across 10 hyperparameter variables for a recommendation model and need an initial screening strategy to find the most influential settings before doing final optimization. Could you construct a comprehensive experimental execution plan covering factor level definitions, screening matrix setup, metric standards, and run count estimation?	fail→fail	22,366	14,594	-35%	1	1	0%	4,237	3,542	-16%	0	0	—
case-03 We ran our benchmark on 4 database configurations and collected execution latency measurements across 100 queries each. Here are the observed latency numbers: Config A = [12, 14, 11], Config B = [22, 25, 21], Config C = [10, 9, 11], Config D = [18, 19, 17]. Can you compute the F-statistic for a one-way ANOVA to test if configuration latency differences are statistically significant?	pass→pass	17,798	17,389	-2%	1	1	0%	4,253	4,882	+15%	0	0	—
case-04 We are running sequential hyperparameter tuning using Gaussian Process regression. Given our previous 5 evaluated points and their validation loss outputs, write a Python function using Expected Improvement (EI) acquisition function to select the single next point to sample.	pass→pass	15,539	14,858	-4%	1	1	0%	3,088	3,648	+18%	0	0	—
case-05 We evaluated 20 different model variants against baseline and obtained the following raw p-values: [0.001, 0.004, 0.012, 0.035, 0.048, 0.08, 0.12, 0.15]. Apply the Benjamini-Hochberg procedure at FDR q=0.05 to determine which hypotheses remain significant after controlling the false discovery rate.	pass→pass	10,870	19,554	+80%	1	1	0%	2,545	5,332	+110%	0	0	—
case-06 We need to optimize a machine learning pipeline with 3 factors (learning rate, weight decay, batch size). We have sufficient compute budget to test all combinations and need to capture all main effects and high-order interaction effects among the factors. Should we use random grid search or a structured experimental design approach?	fail→pass	13,781	11,244	-18%	1	1	0%	2,338	2,733	+17%	0	0	—
case-07 We are evaluating 5 hyperparameters for an object detection model. We need to accurately estimate all main effects and all 2-factor interaction effects, but we cannot afford a full 32-run factorial design. A colleague suggested a Resolution III design with 8 runs to save costs. Which design level should we select to preserve 2-factor interactions?	pass→pass	8,373	9,529	+14%	1	1	0%	1,596	2,400	+50%	0	0	—
case-08 Our engineering team wants to screen 12 different compiler optimization flags to find which ones improve binary execution speed. We only care about main effects during this initial phase. Someone proposed running a Resolution IV fractional factorial design with 32 runs, but compute is constrained. What design method is specifically suited for screening 12 factors, and how many runs does it require?	pass→pass	13,034	6,861	-47%	1	1	0%	2,345	1,918	-18%	0	0	—
case-09 We are designing a hardware stress test across 6 control parameters. The system operates in harsh environments where ambient temperature and power supply voltage fluctuate randomly as noise factors. We need an experimental design method that minimizes output variance against these noise factors rather than just optimizing mean latency. What design approach should we choose?	fail→pass	14,654	10,403	-29%	1	1	0%	2,482	2,452	-1%	0	0	—
case-10 We completed initial factor screening for 4 continuous synthesis parameters and identified them as key drivers. Now we need to build a response surface to find the precise smooth non-linear optimal point including quadratic curvature effects. What experimental design methodology is designed for this post-screening optimization phase?	pass→pass	13,453	7,703	-43%	1	1	0%	2,254	1,977	-12%	0	0	—
case-11 We are setting up an experimental design pipeline from scratch. Our project lead suggested constructing the execution matrix grid first, then picking levels and metrics afterwards. What is the correct sequence of phases from factor identification through sample size estimation?	pass→pass	14,367	12,325	-14%	1	1	0%	2,271	2,715	+20%	0	0	—
case-12 We have 7 hyperparameters for a transformer model and need an initial screening matrix to identify main effects only on a tight budget. We do not need to unconfound 2-factor interactions in this round. What fractional factorial resolution is designed for main-effects-only screening across 5 to 8 factors?	pass→pass	6,820	5,592	-18%	1	1	0%	1,197	1,616	+35%	0	0	—
case-13 We want to screen 9 configuration parameters for a distributed storage system to find main drivers. We want the minimum number of runs possible for screening without doing a high-run fractional factorial. Which design type applies to 8-15 factors, and how many runs will be executed?	pass→pass	8,904	3,994	-55%	1	1	0%	1,560	1,361	-13%	0	0	—
case-14 We are evaluating 5 system settings under variable network latency noise conditions. We want to use a Taguchi robust design approach to minimize variance. What specific Taguchi orthogonal array sizes are typical for 4 to 8 factors?	fail→fail	12,550	7,590	-40%	1	1	0%	2,310	2,099	-9%	0	0	—
case-15 After screening, we are applying a Central Composite Design (CCD) under Response Surface Methodology for 3 tuning factors. What structural components make up the run points in a standard CCD matrix?	pass→pass	9,451	7,076	-25%	1	1	0%	1,710	2,009	+17%	0	0	—
case-16 We constructed our fractional factorial design matrix for 5 model settings. Now we need to determine how many replicate execution runs to perform per cell matrix entry to reliably detect a target effect size. What statistical technique should be used to estimate this count?	pass→pass	12,802	12,813	+0%	1	1	0%	2,188	2,966	+36%	0	0	—
case-17 We have 4 critical algorithms to evaluate in a small sandbox. Computer budget is unlimited for this test, and we must measure every potential high-order multi-way interaction between all 4 variables. What design type should we select, and how many total matrix run combinations will it produce at 2 levels per factor?	pass→pass	5,768	5,603	-3%	1	1	0%	1,042	1,653	+59%	0	0	—
case-18 We are initiating a formal experimental design for evaluating database tuning parameters. What is the very first step in the execution flow before specifying discrete level boundaries or constructing matrices?	pass→pass	9,865	3,427	-65%	1	1	0%	1,543	1,238	-20%	0	0	—
case-19 We need to screen 14 separate environment flags for a CI/CD build pipeline to isolate main effects on build time. Compute budget is severely restricted. Which screening design method handles up to 15 factors with minimal runs?	pass→pass	11,471	8,766	-24%	1	1	0%	2,029	2,184	+8%	0	0	—
case-20 We have 4 parameters to evaluate and need to resolve 2-factor interactions without running a full 16-run factorial. We want a Resolution V design. How many runs does a Resolution V fractional factorial require for 4 factors at 2 levels?	pass→pass	12,160	13,418	+10%	1	1	0%	2,120	3,181	+50%	0	0	—
case-21 During the level specification step for continuous factors in an experimental design, how many discrete levels per factor are typically chosen for standard designs before moving to response surface modeling?	pass→pass	8,666	5,727	-34%	1	1	0%	1,402	1,539	+10%	0	0	—
case-22 Our team is comparing standard full factorial designs against Taguchi orthogonal arrays for a physical manufacturing test. What is the primary functional objective of Taguchi designs regarding noise factors?	pass→pass	7,425	7,052	-5%	1	1	0%	1,192	1,676	+41%	0	0	—

These cases were also screened by the skill’s intent / safety pass.See the Trust review ↗Reproduce this benchmark in Colab ↗

How this skill was measured

DecimalAI ran this skill against gemini-3.6-flash twice over the same eval suite — once with the skill loaded and once without — and compared the two runs case by case. 22 cases were attempted. The headline lift of +14 percentage points is the difference between those two pass rates over the 22 comparable cases.

Model measured: gemini-3.6-flash
Graded by: gemini-3.6-flash
Grading method: judged
Verification: verified
Pass rate without the skill: 77.3%
Pass rate with the skill: 90.9%
Cases attempted: 22
Cases counted toward lift: 22
Cases that regressed: 0
Skill version measured: v1
Run completed: 8/22/2026, 21:32 UTC

Measured example — same prompt, with vs. without the skill · from the benchmark run · case case-01

Prompt

I am preparing an AI fine-tuning experiment where I need to evaluate learning rate, batch size, dropout rate, weight decay, and sequence length simultaneously. Because GPU time is limited, we cannot test every single combination. Can you guide me through identifying the variables, picking a suitable experiment design method given our budget, outlining the level specifications, and detailing how to estimate sample sizes and metrics?

Without the skill loaded, the model failed this case. With it loaded, the same prompt on the same model passed. This is one improved case from the latest verified run; every case, including any that regressed, is in the table above.

Show both answers

SkillScore blends 4 signals1 of 4 complete

BenchmarkPass rate WITH skill on a verified run91

Live evalNeeds ≥20 evaluated traces in 30 days

AI ratingNeeds ≥10 rated traces

AdoptionNeeds ≥10 serves across ≥2 orgs in 30 days

Efficiency · separate axis (at equal correctness)+0% turns+23% tokens

Shown for transparency; not credited unless a verified run shows no correctness regression.

Backed by 1 of 4 signals — each added signal makes the score more robust.

Verified on gemini-3.6-flash · re-tested 6 days ago

Other measured skills in the registry, with their headline benchmark lift.