Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

Benchmark Optimization Loop · affaan-m | DecimalAI

Benchmark Optimization Loop

SkillScore45Maintained

benchmark-optimization-loop

Use when the user asks to make something faster, try many variants, run recursive optimization, benchmark latency/throughput/cost, or choose the best implementation by repeated measured tests.

DevOpsby affaan-m

Links this skill to your agents and keeps it current with the author’s updates.

Automatic

DevOps & SRE pack

1 · One-time — install the CLI (docs ↗)

pip install decimalai

2 · Get the skill — writes .claude/skills/benchmark-optimization-loop/SKILL.md

decimalai skills pull benchmark-optimization-loop --out .claude/skills/

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/benchmark-optimization-loop/SKILL.md https://app.decimal.ai/s/benchmark-optimization-loop/SKILL.md

Raw file: /s/benchmark-optimization-loop/SKILL.md ↗ · machine-readable summary at /s/benchmark-optimization-loop.json ↗

In productionGlobal

how this skill behaves on real traffic · last 30 days

No production data for this skill yet — usage numbers appear here once agents start running it.

Anonymized aggregate across every org using this skill — proof from real production use, not a demo.

Does this skill help? · Benchmark: with vs. withoutsign in to run this on your agents

22 test cases from the publisher’s eval.yaml · each run twice on gemini-3.6-flash — with the skill injected, then without · last run 55d ago

Yes — +41% lift-5% tokens0% turns45% of whole cases pass with skill (10/22)

Test case	Without → With	Effect	Δ tokens	Δ turns
case-07	✗→✓	▲ Improved	—	—
case-11	✗→✓	▲ Improved	—	—
case-16	✗→✓	▲ Improved	—	—
case-05	✗→✓	▲ Improved	—	—
case-04	✗→✓	▲ Improved	—	—

Showing 5 of 22 cases

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v4 — 7/30/2026

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

affaan-m / ECC

skills/benchmark-optimization-loop/SKILL.md

View on GitHub ↗

License MITImported 8/3/2026Version v1

Skill contents

SKILL.md

Benchmark Optimization Loop

Use this skill to convert "make it 20x faster" or "try 50 recursive optimizations" into a bounded measured loop that can actually improve a system.

Required Baseline

Do not optimize until these exist:

the operation being optimized;
the correctness gate that must stay green;
the metric: wall time, p95 latency, rows/sec, cost/run, memory, error rate;
the current baseline;
the search budget: max variants, max time, max spend, max data impact.

If the user asks for an unrealistic target, keep the ambition but make the loop bounded and measurable.

Loop

Measure the baseline.
Identify bottlenecks from evidence.
Generate variants that test one hypothesis each.
Run variants with the same input shape.
Reject variants that fail correctness, safety, or reproducibility.
Promote the fastest safe variant.
Codify the winning path in a script, command, test, config, or doc.
Rerun the baseline and winner to confirm the delta.

Variant Table

Track variants like this:

text
Variant | Hypothesis | Command | Time | Correct? | Notes
baseline | current path | npm run job | 120s | yes | stable
batch-500 | fewer round trips | npm run job -- --batch 500 | 42s | yes | winner
parallel-8 | more workers | npm run job -- --workers 8 | 31s | no | rate limited

Recursive Search

For recursive or hyperparameter work:

persist every run to a ledger;
compare against the prior accepted winner, not only the previous run;
keep a holdout or replay check;
stop when improvement is within noise, correctness fails, cost exceeds the

budget, or the search starts changing more variables than it can explain.

Use phrases like "best measured safe variant" instead of "global optimum" unless the search space was actually exhaustive.

Promotion Gate

A variant cannot become the new default until:

correctness tests pass;
the performance delta is repeated or explained;
rollback is obvious;
the change is encoded in source control or a durable runbook;
the final summary includes exact commands and measurements.

Illustrative Want to see what benchmark-optimization-loop would put in front of your model? Preview the rendered body and check its trigger phrases — no LLM call, no signup.

Latest benchmark · publisher runVerified

7/27/2026, 16:14 UTC · 22 cases

Lift

+41%

45% vs 5%

Agent turns

1.0 vs 1.0

Tokens

-5%

2,108.59 vs 2,214.73

fail → pass · 9pass → pass · 1fail → fail · 12

Per-case results (22)

Case	Status	Duration (ms)			Turns			Tokens			Tool calls
Case	Status	Without	With	Δ	Without	With	Δ	Without	With	Δ	Without	With	Δ
case-01 Our nightly ETL data processing script takes over 4 hours to run, and leadership wants us to make it 20 times faster by morning. I need a structured performance testing process to safely run experiments, along with a tabular layout to log each hypothesis, command, timing, and validation status so we can identify the top performing safe approach.	fail→fail	—	—	—	—	—	—	—	—	—	—	—	—
case-10 When running an iterative performance search on a distributed Spark job, under what specific conditions should the recursive search loop stop automatically?	fail→fail	—	—	—	—	—	—	—	—	—	—	—	—
case-02 I am trying to optimize the throughput of our document embeddings service by tweaking batch sizes and worker concurrency, but previous attempts caused out-of-memory crashes. Can you establish a systematic benchmarking strategy for recursive parameter searches that tracks run history, enforces safety guardrails, and outlines clear criteria before promoting a new configuration to production?	fail→fail	—	—	—	—	—	—	—	—	—	—	—	—
case-13 What documentation artifact must accompany the final approval when promoting an optimized Java JVM garbage collection flags setup?	fail→fail	—	—	—	—	—	—	—	—	—	—	—	—
case-14 Our Python backend service is struggling under heavy traffic. What is the second step in the performance optimization loop once baseline metrics are recorded?	fail→fail	—	—	—	—	—	—	—	—	—	—	—	—
case-07 A Postgres query tuning experiment produced a variant that cut latency by 40% safely. What concrete step must be taken immediately after promoting this winning variant?	fail→pass	—	—	—	—	—	—	—	—	—	—	—	—
case-06 During a performance benchmark run on a Golang gRPC service, one candidate variant yielded a 50% latency drop but occasionally returned stale data during concurrent writes. How should this variant be handled in the optimization loop?	fail→fail	—	—	—	—	—	—	—	—	—	—	—	—
case-11 We completed 20 optimization runs on a Node.js worker pool and found a safe configuration that reduces memory consumption by 30%. What phrasing should be used in the report to describe this result?	fail→pass	—	—	—	—	—	—	—	—	—	—	—	—
case-08 We finished testing five configuration changes on an Nginx load balancer and selected the best performing candidate. How do we conclusively verify the performance delta before concluding the loop?	pass→pass	—	—	—	—	—	—	—	—	—	—	—	—
case-16 We need to set up a standardized markdown log for our Rust binary optimization benchmarks. What exact column headers belong in the variant tracking table?	fail→pass	—	—	—	—	—	—	—	—	—	—	—	—
case-05 We are benchmarking a Redis cache layer for a high-traffic microservice. How should optimization variants be generated and structured during each loop iteration?	fail→pass	—	—	—	—	—	—	—	—	—	—	—	—
case-19 A memory allocation variant in a C++ network driver yielded a 15% speedup on trial 1, but trial 2 showed only a 2% gain. Can this variant be promoted immediately?	fail→fail	—	—	—	—	—	—	—	—	—	—	—	—
case-04 We want to speed up an API endpoint that handles SQL queries using an optimization loop. What specific baseline prerequisites must be established before running any performance experiments?	fail→pass	—	—	—	—	—	—	—	—	—	—	—	—
case-22 How do I write a multi-stage Dockerfile for a Go application to minimize final image size using scratch as the base image?	fail→fail	—	—	—	—	—	—	—	—	—	—	—	—
case-18 During iteration 12 of a compiler optimization search, should candidate 12 be evaluated against iteration 11 or against something else?	fail→fail	—	—	—	—	—	—	—	—	—	—	—	—
case-09 We are running an automated hyperparameter search over machine learning pipeline compilation flags. What rule governs how trial runs are recorded during recursive tuning?	fail→pass	—	—	—	—	—	—	—	—	—	—	—	—
case-12 We have a candidate variant that speeds up Docker container image build times by 4x. What architectural criteria must be satisfied before this variant becomes the new production default?	fail→pass	—	—	—	—	—	—	—	—	—	—	—	—
case-03 Our web application's test suite execution time has bloated to 35 minutes, hurting developer productivity. I want to systematically benchmark different parallelization and caching strategies. Please give me an iterative benchmark loop workflow and a promotion checklist to safely evaluate, verify, and document performance improvements.	fail→pass	—	—	—	—	—	—	—	—	—	—	—	—
case-21 Write the PostgreSQL DDL command to create a composite B-tree index on the orders table for columns tenant_id and created_at.	fail→fail	—	—	—	—	—	—	—	—	—	—	—	—
case-15 Our CTO asked us to reduce our cloud Kubernetes cluster compute costs by 95% within 24 hours. How should an optimization loop respond to this directive?	fail→pass	—	—	—	—	—	—	—	—	—	—	—	—
case-17 When evaluating three distinct caching variants for an Elasticsearch cluster, how should the evaluation workload inputs be structured across runs?	fail→fail	—	—	—	—	—	—	—	—	—	—	—	—
case-20 We are experiencing severe CPU throttling on our Linux kernel web server. Which flamegraph visualization tools (such as perf, bpftrace, or py-spy) should we install to identify top stack frames?	fail→fail	—	—	—	—	—	—	—	—	—	—	—	—

These cases were also screened by the skill’s intent / safety pass.See the Trust review ↗Reproduce this benchmark in Colab ↗

How this skill was measured

DecimalAI ran this skill against gemini-3.6-flash twice over the same eval suite — once with the skill loaded and once without — and compared the two runs case by case. 22 cases were attempted. The headline lift of +41 percentage points is the difference between those two pass rates over the 22 comparable cases.

Model measured: gemini-3.6-flash
Graded by: gemini-3.6-flash
Grading method: judged
Verification: verified
Pass rate without the skill: 4.6%
Pass rate with the skill: 45.5%
Cases attempted: 22
Cases counted toward lift: 22
Cases that regressed: 0
Skill version measured: v1
Run completed: 7/27/2026, 16:14 UTC

The per-case answers from this run were removed by the retention sweep, so the case table below shows the verdicts without the text either arm produced. The counts above were recorded at the time and are unaffected. Answers are now kept for 180 days.

SkillScore blends 4 signals1 of 4 complete

BenchmarkPass rate WITH skill on a verified run45

Live evalNeeds ≥10 evaluated traces in 30 days

AI ratingNeeds ≥10 rated traces

AdoptionNeeds ≥10 serves across ≥2 orgs in 30 days

Efficiency · separate axis (at equal correctness)+0% turns-15% tokens

Shown for transparency; not credited unless a verified run shows no correctness regression.

Backed by 1 of 4 signals — each added signal makes the score more robust.

Verified on gemini-3.6-flash · re-tested 55 days ago

Other measured skills in the registry, with their headline benchmark lift.

Ready to supercharge your AI agents?

Benchmark Optimization Loop

SkillSafety

Source

Benchmark Optimization Loop

Required Baseline

Loop

Variant Table

Recursive Search

Promotion Gate

How this skill was measured

Related skills