Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

WeaveBench CUA-Harness Reproduction · amap-ml | DecimalAI

amap-ml/weavebench-cua-reproduce

SkillScore57Maintained

Reproduce CUA-Harness experiments on WeaveBench from a GitHub checkout. Use when the user wants an AI coding agent to set up dependencies, download WeaveBench assets, prepare the 120G VM, configure Qwen/Anthropic-compatible APIs, run smoke tests, launch full or subset evaluations, inspect logs, or summarize scores for this repository.

AI/MLby amap-ml

Links this skill to your agents and keeps it current with the author’s updates.

Automatic

AI Engineer pack

1 · One-time — install the CLI (docs ↗)

pip install decimalai

2 · Get the skill — writes .claude/skills/amap-ml-weavebench-cua-reproduce/SKILL.md

decimalai skills pull amap-ml-weavebench-cua-reproduce --out .claude/skills/

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/amap-ml-weavebench-cua-reproduce/SKILL.md https://app.decimal.ai/s/amap-ml-weavebench-cua-reproduce/SKILL.md

Raw file: /s/amap-ml-weavebench-cua-reproduce/SKILL.md ↗ · machine-readable summary at /s/amap-ml-weavebench-cua-reproduce.json ↗

In productionGlobal

how this skill behaves on real traffic · last 30 days

No production data for this skill yet — usage numbers appear here once agents start running it.

Anonymized aggregate across every org using this skill — proof from real production use, not a demo.

Does this skill help? · Benchmark: with vs. withoutsign in to run this on your agents

23 test cases from the publisher’s eval.yaml · each run twice on gemini-3.6-flash — with the skill injected, then without · last run 28d ago

—+65% tokens0% turns57% of whole cases pass with skill (13/23)

Test case	Without → With	Effect	Δ tokens	Δ turns
case-06	✗→✓	▲ Improved	-36%	0%
case-04	✗→✓	▲ Improved	1410%	0%
case-08	✗→✓	▲ Improved	140%	0%
case-10	✗→✓	▲ Improved	180%	0%
case-11	✗→✓	▲ Improved	19%	0%

Showing 5 of 23 cases

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v4 — 8/10/2026

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

amap-ml / longhorizon-harness

eval/WeaveBench-harness/skills/weavebench-cua-reproduce/SKILL.md

View on GitHub ↗

License MITImported 8/10/2026Version v1Bundled 5 files · scripts

Skill contents

scripts/

references/

SKILL.md

WeaveBench CUA-Harness Reproduction

Use this skill to guide an agent through a complete, reproducible WeaveBench CUA-Harness run. The expected repository layout is:

text
WeaveBench-harness/
  WeaveBench/
  cua-harness/
  skills/weavebench-cua-reproduce/

Do not invent alternate launch commands. Prefer the bundled helper script and the project scripts under WeaveBench/scripts/.

Supported Scope

Supported automation:

Local Docker/KVM WeaveBench evaluation.
Qwen 3.7-Plus through an Anthropic-compatible endpoint.
Claude Code execution backend through cua_harness_claudecode.
OpenClaw judge setup and verification.
Official WeaveBench assets downloaded from the HuggingFace dataset.
A 120G copy of the official Ubuntu qcow2 plus per-task rootfs growth.

Not automated:

Cloud VM provisioning.
Public image-hosting service deployment.
New model-provider adapters beyond the existing Anthropic-compatible path.
Non-Claude-Code execution backend reproduction.
Deleting or pruning existing results.

If the user requests unsupported automation, explain the boundary and provide the closest supported local Docker/KVM path.

Start With Intake

Before doing setup work on a new machine, run or mentally follow:

bash
./skills/weavebench-cua-reproduce/scripts/reproduce.sh intake

Confirm:

Linux Docker/KVM is available or installable.
The user accepts large downloads and creation of Ubuntu_120G.qcow2.
The model endpoint and API key are available.
Image proxy should be enabled with public upload/show URLs, or disabled because the endpoint accepts large base64 screenshots.
The desired scope is doctor, smoke, subset, or full 114-task evaluation.
Long-running/full evaluation is allowed, especially if other experiments may already be active.

Workflow

Locate the project root. It must contain both WeaveBench/ and cua-harness/.
Read references/configuration.md when you need exact defaults, environment variables, or paths.
Read references/assets.md before downloading or validating WeaveBench tasks, runtime assets, judge templates, or VM files.
Read references/verify.md before reporting whether setup or a run is actually verified.
Read references/troubleshooting.md when setup, VM, API, image proxy, warmup, or judge errors occur.
Ask the user for missing API details only when they are required and not discoverable from the environment.
Run commands from the project root unless a command explicitly changes into WeaveBench/.

Helper Script

Use:

bash
./skills/weavebench-cua-reproduce/scripts/reproduce.sh <command>

Commands:

text
install     Install local WeaveBench package and OpenClaw if needed.
download    Download task files, Claude Code runtime, judge template, and VM.
vm120g      Create cache/vm/Ubuntu_120G.qcow2 from the official Ubuntu.qcow2.
doctor      Run the read-only environment checker.
intake      Print setup questions for a new machine.
status      Print a non-destructive setup status report.
smoke       Run a one-task, one-round smoke test.
full        Launch the full 114-task evaluation in tmux.
stats       Summarize scores for a result directory.
plan        Print the minimal manual command sequence.

For a fresh machine, the usual sequence is:

bash
./skills/weavebench-cua-reproduce/scripts/reproduce.sh install
./skills/weavebench-cua-reproduce/scripts/reproduce.sh download
./skills/weavebench-cua-reproduce/scripts/reproduce.sh vm120g
./skills/weavebench-cua-reproduce/scripts/reproduce.sh status
./skills/weavebench-cua-reproduce/scripts/reproduce.sh doctor
./skills/weavebench-cua-reproduce/scripts/reproduce.sh smoke
./skills/weavebench-cua-reproduce/scripts/reproduce.sh full

Required User Configuration

Before doctor, smoke, or full, ensure these are set:

bash
export WEAVEBENCH_LITELLM_KEY="YOUR_API_KEY"
export WEAVEBENCH_LITELLM_BASE_URL="https://YOUR_ANTHROPIC_COMPATIBLE_ENDPOINT/v1"

If the provider cannot accept large base64 image payloads, enable image URL proxy:

bash
export WEAVEBENCH_IMAGE_PROXY=1
export WEAVEBENCH_IMAGE_PROXY_UPLOAD_URL_TPL="https://YOUR_IMAGE_UPLOAD_ENDPOINT/{id}"
export WEAVEBENCH_IMAGE_PROXY_SHOW_URL_TPL="https://YOUR_PUBLIC_IMAGE_URL/{id}.png"
export WEAVEBENCH_IMAGE_PROXY_UPLOAD_MODE="raw"

If the provider can directly accept large base64 screenshots:

bash
export WEAVEBENCH_IMAGE_PROXY=0

Launch Rules

Use the 120G VM copy by default: WeaveBench/cache/vm/Ubuntu_120G.qcow2.
Keep WEAVEBENCH_GROW_ROOTFS=1 unless the user explicitly disables it.
Use qwen3.7-plus, cua_harness_claudecode, GUI mode, 25 CUA rounds, and 5 concurrent VM environments for full evaluation unless the user asks for a subset.
Use smoke before full on a new machine.
Run long evaluations in tmux.
Never delete cache/, results/, logs/, Docker volumes, or judge workspaces unless the user explicitly asks.
When reporting API keys, redact them unless the user explicitly requests full values.
At the end of setup, classify each surface as configured and verified, configured but not smoke-tested, not configured, or blocked awaiting user action.

Common Subsets

Run one domain:

bash
export WEAVEBENCH_DOMAINS="WEB"
./skills/weavebench-cua-reproduce/scripts/reproduce.sh full

Run one task:

bash
export WEAVEBENCH_DOMAINS="WEB"
export WEAVEBENCH_TASK_FILTER="WEB_task_10_lighthouse"
export WEAVEBENCH_NUM_ENVS=1
./skills/weavebench-cua-reproduce/scripts/reproduce.sh full

Summarize a run:

bash
./skills/weavebench-cua-reproduce/scripts/reproduce.sh stats \
  WeaveBench/results/<run_name>/gui/qwen3.7-plus

Final Report Template

End setup or launch tasks with this shape:

text
Environment: configured and verified / configured but not smoke-tested / not configured / blocked awaiting user action
Assets: ...
120G VM: ...
Model API: ...
Image proxy: ...
Judge: ...
Smoke test: passed / failed / skipped
Full run: launched in tmux <name> / not launched
Provenance: run_provenance.json written / not applicable
Remaining blockers: ...

Illustrative Want to see what amap-ml/weavebench-cua-reproduce would put in front of your model? Preview the rendered body and check its trigger phrases — no LLM call, no signup.

Latest benchmark · publisher runVerified

8/31/2026, 04:21 UTC · 23 cases

Measured on 17 comparable cases — below the 22 needed to publish a lift, so this result is shown here but not on the registry card or leaderboard.

Lift

+43%

57% vs 13%

Agent turns

1.0 vs 1.0

Tokens

+65%

3,030.06 vs 1,833.94

fail → pass · 13fail → fail · 7pass → fail · 3

Per-case results (23)

Case	Status	Duration (ms)			Turns			Tokens			Tool calls
Case	Status	Without	With	Δ	Without	With	Δ	Without	With	Δ	Without	With	Δ
case-06 I have access to a proprietary Gemini 1.5 Pro endpoint that requires Google Cloud ADC authentication instead of an Anthropic-compatible API. Can you set up the model provider adapter for this and run the WeaveBench doctor?	fail→pass	26,801	10,347	-61%	1	1	0%	4,056	2,580	-36%	0	0	—
case-01 I have a fresh Ubuntu machine and need to get the WeaveBench harness fully set up. Please walk through your standard initial checks, grab all required assets, prep the local virtual machine disk, and perform a preliminary test run to confirm it is functional. Conclude by providing the standard summary report detailing the environment's configuration state, the status of the VM and assets, the model API and proxy configurations, judge status, the result of the quick test, and any remaining blockers.	fail→fail	12,476	9,330	-25%	1	1	0%	492	1,972	+301%	0	0	—
case-02 We are ready to execute the entire suite of evaluation tasks. Please launch the full run, ensuring you use the appropriate method to keep the terminal process alive in the background for a long duration. When you are done initiating it, output the final setup template showing whether the environment is verified, the status of the model API and image proxy, the judge setup, and the background session name for the newly launched run.	fail→fail	8,319	9,785	+18%	1	1	0%	233	1,997	+757%	0	0	—
case-03 Could you automate the provisioning of a cloud server and deploy an external image hosting service for my WeaveBench tests? If those actions are outside your supported scope, please explain the limitations and instead configure the closest local Docker-based alternative for just the WEB domain. Execute the setup steps and provide the mandated final readout covering the environment status, asset downloads, model API, proxy details, and any blockers awaiting my input.	pass→fail	10,719	15,618	+46%	1	1	0%	2,023	2,121	+5%	0	0	—
case-04 I want to run a WeaveBench CUA test but using the newly released 'OpenHands' backend instead of the default. Can you configure the reproduction suite to use OpenHands as the execution backend and start a smoke test?	fail→pass	15,011	12,033	-20%	1	1	0%	206	3,111	+1410%	0	0	—
case-05 My disk space is getting full from previous WeaveBench runs. Before launching a new full evaluation, can you prune all existing results in the results/ and cache/ directories?	fail→fail	4,635	18,095	+290%	1	1	0%	705	2,172	+208%	0	0	—
case-07 I'm starting on a completely new Linux host for the WeaveBench-harness. I want to verify if the machine is ready. Should I run the doctor script first, or something else? I don't want to create any files yet.	fail→fail	7,895	15,959	+102%	1	1	0%	1,335	3,019	+126%	0	0	—
case-08 I am ready to download the WeaveBench assets. Should I cd into the WeaveBench subdirectory first to run the download scripts?	fail→pass	10,628	3,669	-65%	1	1	0%	997	2,394	+140%	0	0	—
case-09 What is the exact sequence of helper script commands I should execute to go from a totally fresh machine to a running full evaluation? Please list them out in order.	fail→fail	8,993	10,336	+15%	1	1	0%	1,418	2,791	+97%	0	0	—
case-10 I need to set up my API keys for the WeaveBench Anthropic-compatible endpoint. Normally I just set `OPENAI_API_KEY` and `OPENAI_BASE_URL`. What exact variables does this reproduction environment require before I run the doctor command?	fail→pass	17,926	9,933	-45%	1	1	0%	954	2,674	+180%	0	0	—
case-11 My model endpoint is capable of receiving large base64 screenshots directly without choking. Should I just leave the proxy variables unset, or is there a specific way to disable the image proxy?	fail→pass	17,769	8,790	-51%	1	1	0%	2,032	2,423	+19%	0	0	—
case-12 My provider rejects large base64 image payloads. I will use a local image proxy. I know I need to set `WEAVEBENCH_IMAGE_PROXY=1` and the upload/show URL templates. Does the proxy require any specific upload mode?	fail→pass	15,959	16,617	+4%	1	1	0%	2,043	2,478	+21%	0	0	—
case-13 I downloaded the official Ubuntu.qcow2 image for WeaveBench. Do I need to manually resize it using qemu-img to get the 120G capacity, or does the helper script do that?	fail→pass	29,975	9,807	-67%	1	1	0%	1,858	2,633	+42%	0	0	—
case-14 I want to launch the full 114-task WeaveBench evaluation. I assume I should use GPT-4o, text mode, 10 CUA rounds, and 1 concurrent VM to be safe. How should I configure this before running the full script?	fail→fail	19,846	28,166	+42%	1	1	0%	2,486	4,890	+97%	0	0	—
case-15 When running the WeaveBench evaluation, I notice the disk fills up. Should I set `WEAVEBENCH_GROW_ROOTFS=0` to prevent excessive disk usage?	fail→pass	15,246	8,997	-41%	1	1	0%	1,816	3,451	+90%	0	0	—
case-16 I only want to run the WeaveBench evaluation for the 'WEB' domain. Do I pass '--domain WEB' to the full run command?	fail→pass	5,903	8,156	+38%	1	1	0%	1,006	2,361	+135%	0	0	—
case-17 I want to run a specific task: 'WEB_task_10_lighthouse'. How do I isolate this task using the environment variables? Should I leave the number of environments at 5?	fail→pass	16,979	11,545	-32%	1	1	0%	1,882	3,041	+62%	0	0	—
case-18 The run finished. I have the results in `WeaveBench/results/run_xyz/gui/qwen3.7-plus`. How do I generate a score summary? Is there a python script I need to call directly?	fail→pass	15,000	12,809	-15%	1	1	0%	1,774	3,013	+70%	0	0	—
case-19 My setup is failing. Here is my status report. Can you print out the environment variables including the LiteLLM API key so I can verify they are being read correctly?	pass→fail	11,401	14,211	+25%	1	1	0%	1,018	4,323	+325%	0	0	—
case-20 I've run the setup scripts, but I want to be absolutely sure the run is verified before I report back to my team. What is the criteria for a verified WeaveBench setup?	fail→fail	20,422	13,669	-33%	1	1	0%	2,124	3,191	+50%	0	0	—
case-21 I want to manually download the WeaveBench judge templates and validate the VM files before running any scripts. Which file explains the expected checksums and paths?	fail→pass	15,763	10,418	-34%	1	1	0%	2,011	2,711	+35%	0	0	—
case-22 I'm encountering an issue during the warmup phase where the image proxy throws a 502 error. Where can I find information on fixing proxy and warmup errors?	fail→pass	17,407	15,030	-14%	1	1	0%	2,367	3,538	+49%	0	0	—
case-23 I have finished configuring the model API and downloaded the assets for WeaveBench, but I haven't run the smoke test yet. Please output the setup summary report.	pass→fail	13,824	10,808	-22%	1	1	0%	1,157	2,386	+106%	0	0	—

These cases were also screened by the skill’s intent / safety pass.See the Trust review ↗Reproduce this benchmark in Colab ↗

How this skill was measured

DecimalAI ran this skill against gemini-3.6-flash twice over the same eval suite — once with the skill loaded and once without — and compared the two runs case by case. 23 cases were attempted, and 17 counted toward the lift figure. The other 6 produced results that are not comparable between the two arms, so they are excluded from the headline rather than averaged into it. The headline lift of +43 percentage points is the difference between those two pass rates over the 17 comparable cases. 4 cases got worse with the skill loaded, and they are included in that figure.

Model measured: gemini-3.6-flash
Graded by: gemini-3.6-flash
Grading method: judged
Verification: verified
Pass rate without the skill: 13%
Pass rate with the skill: 56.5%
Cases attempted: 23
Cases counted toward lift: 17
Cases that regressed: 4
Skill version measured: v1
Run completed: 8/31/2026, 04:21 UTC

Measured example — same prompt, with vs. without the skill · from the benchmark run · case case-06

Prompt

I have access to a proprietary Gemini 1.5 Pro endpoint that requires Google Cloud ADC authentication instead of an Anthropic-compatible API. Can you set up the model provider adapter for this and run the WeaveBench doctor?

Without the skill loaded, the model failed this case. With it loaded, the same prompt on the same model passed. This is one improved case from the latest verified run; every case, including any that regressed, is in the table above.

Show both answers

SkillScore blends 4 signals1 of 4 complete

BenchmarkPass rate WITH skill on a verified run57

Live evalNeeds ≥10 evaluated traces in 30 days

AI ratingNeeds ≥10 rated traces

AdoptionNeeds ≥10 serves across ≥2 orgs in 30 days

Backed by 1 of 4 signals — each added signal makes the score more robust.

Verified on gemini-3.6-flash · re-tested 28 days ago

Other measured skills in the registry, with their headline benchmark lift.