Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

Get Started Free →

Tactic: Budget-Constrained Design · yogsoth-ai | DecimalAI

yogsoth-ai/budget-constrained-design

SkillScore73Maintained

Optimize experiment design under compute and time budget constraints

scienceby yogsoth-ai

Links this skill to your agents and keeps it current with the author’s updates.

Automatic

AI Engineer pack

1 · One-time — install the CLI (docs ↗)

pip install decimalai

2 · Get the skill — writes .claude/skills/yogsoth-ai-budget-constrained-design/SKILL.md

decimalai skills pull yogsoth-ai-budget-constrained-design --out .claude/skills/

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/yogsoth-ai-budget-constrained-design/SKILL.md https://app.decimal.ai/s/yogsoth-ai-budget-constrained-design/SKILL.md

Raw file: /s/yogsoth-ai-budget-constrained-design/SKILL.md ↗ · machine-readable summary at /s/yogsoth-ai-budget-constrained-design.json ↗

In productionGlobal

how this skill behaves on real traffic · last 30 days

No production data for this skill yet — usage numbers appear here once agents start running it.

Anonymized aggregate across every org using this skill — proof from real production use, not a demo.

Does this skill help? · Benchmark: with vs. withoutsign in to run this on your agents

22 test cases from the publisher’s eval.yaml · each run twice on gemini-3.6-flash — with the skill injected, then without · last run 7d ago

Yes — +18% lift-1% tokens0% turns73% of whole cases pass with skill (16/22)

Test case	Without → With	Effect	Δ tokens	Δ turns
case-02	✗→✓	▲ Improved	7%	0%
case-05	✗→✓	▲ Improved	-17%	0%
case-07	✗→✓	▲ Improved	-6%	0%
case-20	✗→✓	▲ Improved	15%	0%
case-06	✓→✓	＝ Same ✓	-19%	0%

Showing 5 of 22 cases

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v4 — 8/3/2026

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

yogsoth-ai / de-anthropocentric-research-engine

skills/budget-constrained-design/SKILL.md

View on GitHub ↗

License Apache-2.0Imported 8/3/2026Version v1

Skill contents

SKILL.md

Tactic: Budget-Constrained Design

Orchestration Pattern

Assess Budget → Determine available GPU-hours, wall-clock time, and cost ceiling
factor-identification → Identify all candidate factors
Estimate Cost Per Run → Calculate time/compute for a single experiment run
Compute Maximum Runs → budget / cost_per_run = max feasible runs
level-specification → Constrain levels to fit within run budget
Select Design Type → Choose most information-efficient design for the budget
design-matrix-construction → Build the constrained design matrix

Decision Criteria

| Available Runs | Recommended Approach | |---------------|---------------------| | < 10 | One-factor-at-a-time or Plackett-Burman screening | | 10-30 | Fractional factorial (Resolution III-IV) | | 30-60 | Fractional factorial (Resolution V) or Taguchi | | 60-120 | Full factorial on top factors + screening on rest | | 120+ | Full factorial or RSM with replication |

Optimization Strategies

Sequential Design: Run screening first, then detailed study on important factors
Adaptive Allocation: Allocate more runs to high-variance conditions
Early Stopping: Define stopping criteria for clearly dominated configurations
Transfer from Pilot: Use pilot study results to inform main study design
Shared Controls: Reuse control/baseline runs across multiple comparisons

Quality Checks

Does the design have sufficient power for the primary hypothesis?
Are the most important factors given priority in the allocation?
Is there a contingency plan if budget is cut mid-experiment?
Are early stopping criteria pre-defined (not post-hoc)?
Is the design balanced despite budget constraints?

Available SOPs

Optional, no fixed order; the final leaf is always a sop.

| SOP | When to use | | --- | --- | | design-matrix-construction | Build the experiment design matrix with proper orthogonality and balance | | factor-identification | Identify independent, dependent, and control variables for an experiment | | level-specification | Determine appropriate levels for each experimental factor |

Want to see if yogsoth-ai/budget-constrained-design would activate on your phrasing? Try it before you install — no LLM call, no signup.

Latest benchmark · publisher runVerified

8/22/2026, 03:50 UTC · 22 cases

Lift

+18%

73% vs 55%

Agent turns

0%

1.0 vs 1.0

Tokens

-1%

2,456.91 vs 2,473.73

fail → pass · 4pass → pass · 12fail → fail · 6

Per-case results (22)

Case	Status	Duration (ms)			Turns			Tokens			Tool calls
Case	Status	Without	With	Δ	Without	With	Δ	Without	With	Δ	Without	With	Δ
case-01 I need to run a hyperparameter tuning campaign for an LLM fine-tuning project. I have identified 5 independent parameters (learning rate, batch size, warmup steps, weight decay, and LoRA rank). My total compute budget is capped at 200 GPU-hours, and preliminary tests show that a single full training run takes about 8 GPU-hours. Please guide me through a structured design plan to maximize information gain under this constraint, including assessing my run capacity, choosing an appropriate experimental design strategy, and outlining quality checks.	fail→fail	20,503	13,680	-33%	1	1	0%	3,522	3,074	-13%	0	0	—
case-02 I am benchmarking a new RAG system across 4 parameters (chunk size, retrieval depth, prompt template variant, reranking threshold). Due to tight API cost limits, I can only afford to run 6 total execution runs in this phase. What experimental design methodology should I use for such a tiny budget, and how should I structure the screening strategy to get meaningful insights?	fail→pass	18,648	15,512	-17%	1	1	0%	3,028	3,230	+7%	0	0	—
case-03 We are tuning a Vision Transformer model with 6 candidate hyperparameters. Our cloud grant gives us 500 GPU-hours, and each evaluation run takes 10 GPU-hours. Most engineers suggest jumping straight into Bayesian Optimization or random search. What classical experimental design structure fits this exact run capacity to yield systematic factor screening?	fail→fail	14,972	12,036	-20%	1	1	0%	2,580	2,521	-2%	0	0	—
case-04 Our team is optimizing a quantized neural network across 3 core compression factors. We have 1,000 GPU-hours available and each run takes 10 GPU-hours, giving us 100 total runs. Should we do a standard 2-level full factorial or a different structured allocation?	fail→fail	14,581	13,408	-8%	1	1	0%	2,417	2,789	+15%	0	0	—
case-05 We have 300 available runs budgeted for a redis caching parameter tuning experiment. We want to test 3 continuous parameters. Base models often default to simple Latin Hypercube sampling. What experimental design methodology should be recommended for a budget exceeding 120 runs?	fail→pass	16,511	10,098	-39%	1	1	0%	2,550	2,114	-17%	0	0	—
case-06 Please outline the step-by-step procedure for setting up an experiment where GPU budget is severely constrained. Outline the initial step required before factors are constrained or design matrices are constructed.	pass→pass	16,919	9,153	-46%	1	1	0%	2,476	2,008	-19%	0	0	—
case-07 In an experiment with 8 potential parameters where compute is limited to 15 total runs, when in the planning sequence should parameter levels be finalized? Provide the sequence positioning relative to computing run limits and building the final matrix.	fail→pass	11,971	6,404	-47%	1	1	0%	1,712	1,604	-6%	0	0	—
case-08 We have 40 total runs budgeted for testing 12 potential hyperparameters in an inference optimization pipeline. Rather than running all 12 parameters in one big design matrix, how should we structure the progression of experiments across stages to save budget?	pass→pass	19,023	13,011	-32%	1	1	0%	2,838	2,655	-6%	0	0	—
case-09 During an ML hyperparameter exploration with tight compute limits, preliminary runs show that certain learning rate ranges exhibit high variance while others are flat. How should remaining execution runs be reallocated across parameter regions?	pass→pass	18,309	12,313	-33%	1	1	0%	2,357	2,352	-0%	0	0	—
case-10 We want to save compute during an automated architecture search across 30 candidate configurations. To prevent post-hoc bias, when and how should early stopping criteria be established?	pass→pass	15,742	11,278	-28%	1	1	0%	2,227	2,153	-3%	0	0	—
case-11 We completed a 5-run pilot test for a database indexing benchmark before launching our main 40-run experiment. What specific optimization technique uses these pilot results to refine the main experiment layout?	pass→pass	24,547	4,735	-81%	1	1	0%	1,916	1,258	-34%	0	0	—
case-12 We are running 3 separate sub-experiments comparing different model pruning techniques against a baseline BERT architecture under strict compute constraints. How can we minimize baseline run overhead across sub-experiments?	pass→pass	15,771	11,809	-25%	1	1	0%	2,472	2,459	-1%	0	0	—
case-13 What risk management check should be verified regarding hardware preemptions or mid-study budget cuts before executing a cloud experiment?	pass→pass	11,713	6,638	-43%	1	1	0%	1,740	1,451	-17%	0	0	—
case-14 When trimming down an experiment design matrix to fit within a tight 16-run limit, what key statistical property regarding factor distribution must be verified during quality checks?	pass→pass	12,432	8,918	-28%	1	1	0%	1,880	1,861	-1%	0	0	—
case-15 Before launching a budget-constrained model comparison test limited to 12 runs, what statistical check must be performed regarding hypothesis sensitivity?	fail→fail	9,444	5,321	-44%	1	1	0%	1,467	1,359	-7%	0	0	—
case-16 I need to screen 7 binary parameters for a data ingestion pipeline. I have a hard ceiling of 8 execution runs. Base AI assistants often suggest random sampling. What specific classical screening design accommodates 7 factors in 8 runs?	pass→pass	11,997	14,797	+23%	1	1	0%	2,146	3,254	+52%	0	0	—
case-17 We have budget for 20 runs to test 5 factors in a speech-to-text benchmark. Should we aim for Resolution V fractional factorial or Resolution III/IV fractional factorial?	fail→fail	13,154	10,436	-21%	1	1	0%	2,148	2,264	+5%	0	0	—
case-18 We are conducting 45 training runs for an image classification task across 5 factors. What specific resolution level of fractional factorial design is suitable for this run count?	pass→pass	14,395	6,584	-54%	1	1	0%	2,371	1,601	-32%	0	0	—
case-19 A cluster evaluation project has a total cost ceiling of $1,200. Preliminary benchmarking indicates that a single benchmark trial costs $40 in compute credits. How is the maximum number of feasible experiment runs calculated and what is the exact run count?	pass→pass	6,031	4,023	-33%	1	1	0%	897	1,266	+41%	0	0	—
case-20 We have an unlimited GPU budget provided by an enterprise grant to test 3 hyperparameters (learning rate, weight decay, dropout) for a transformer model. We want to map the complete response surface with full cubic polynomial fits and high statistical precision without any run constraints. How should we configure a Response Surface Methodology central composite design?	fail→pass	31,032	30,347	-2%	1	1	0%	5,386	6,174	+15%	0	0	—
case-21 We already collected 500 production telemetry logs from an existing web server cluster running variable worker counts and memory allocations. We did not run a planned experiment; this is historical observational data. How should we analyze factor effects using multiple linear regression and control for confounding variables?	pass→pass	20,174	17,773	-12%	1	1	0%	3,223	3,357	+4%	0	0	—
case-22 We ran a completed experiment comparing Model A and Model B accuracy across 50 evaluations. Model A achieved 84% accuracy and Model B achieved 87%. Assuming a standard alpha of 0.05, how do we calculate the p-value using a two-sample t-test to determine if the difference is statistically significant?	fail→fail	16,178	14,028	-13%	1	1	0%	3,069	3,248	+6%	0	0	—

These cases were also screened by the skill’s intent / safety pass.See the Trust review ↗Reproduce this benchmark in Colab ↗

How this skill was measured

DecimalAI ran this skill against gemini-3.6-flash twice over the same eval suite — once with the skill loaded and once without — and compared the two runs case by case. 22 cases were attempted. The headline lift of +18 percentage points is the difference between those two pass rates over the 22 comparable cases.

Model measured: gemini-3.6-flash
Graded by: gemini-3.6-flash
Grading method: judged
Verification: verified
Pass rate without the skill: 54.6%
Pass rate with the skill: 72.7%
Cases attempted: 22
Cases counted toward lift: 22
Cases that regressed: 0
Skill version measured: v1
Run completed: 8/22/2026, 03:50 UTC

Measured example — same prompt, with vs. without the skill · from the benchmark run · case case-02

Prompt

I am benchmarking a new RAG system across 4 parameters (chunk size, retrieval depth, prompt template variant, reranking threshold). Due to tight API cost limits, I can only afford to run 6 total execution runs in this phase. What experimental design methodology should I use for such a tiny budget, and how should I structure the screening strategy to get meaningful insights?

Without the skill loaded, the model failed this case. With it loaded, the same prompt on the same model passed. This is one improved case from the latest verified run; every case, including any that regressed, is in the table above.

Show both answers

SkillScore blends 4 signals1 of 4 complete

BenchmarkPass rate WITH skill on a verified run73

Live evalNeeds ≥20 evaluated traces in 30 days

AI ratingNeeds ≥10 rated traces

AdoptionNeeds ≥10 serves across ≥2 orgs in 30 days

Efficiency · separate axis (at equal correctness)+0% turns-3% tokens

Shown for transparency; not credited unless a verified run shows no correctness regression.

Backed by 1 of 4 signals — each added signal makes the score more robust.

Verified on gemini-3.6-flash · re-tested 7 days ago

Other measured skills in the registry, with their headline benchmark lift.