Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

gui — cross-driver GUI actuation · svd-ai-lab | DecimalAI

scienceby svd-ai-lab

gui — cross-driver GUI actuation

svd-ai-lab/gui

Source svd-ai-lab/sim-cli ↗License Apache-2.0

Cross-driver GUI actuation for CAE solvers running under sim-cli. Use to click buttons, fill fields, dismiss dialogs, and capture window screenshots against GUI-capable driver windows through `sim exec`.

SkillScore100Maintained

Automatic

Coding pack

Last verified 9/26/2026

What it changes

Passed 22 of 22 test cases with the skill, and 3 of 22 without it.

Without skill14%

With skill100%

Checks it passed that the model alone missed

Calls gui.snapshot(max_depth=4).
Reads the gui.process_filter attribute.
Calls .click("确定") on the window object.

and 28 more in the full report

19 fixed3 passed either way0 still failing0 broke

Pass-rate lift: +86 pp
Tokens: +68%
Agent turns: 0%
Response time: -52%

Model gemini-3.6-flashMeasured 9/26/2026v122 of 22 cases passed with skill

Tokens, turns and time describe resource use, not a dollar price.

Use this skill

Save it into your project's skills folder. Claude Code picks it up in its next session.

curl -fsSL --create-dirs -o .claude/skills/svd-ai-lab-gui/SKILL.md https://app.decimal.ai/s/svd-ai-lab-gui/SKILL.md

Want it pinned and updatable? pip install "decimalai>=0.15", then decimalai skills add svd-ai-lab-gui, and start Claude Code with claude --plugin-dir "$(git rev-parse --show-toplevel)/.agents"

1 · One-time — install the CLI (docs ↗)

pip install "decimalai>=0.15"

2 · Get the skill — writes .agents/skills/svd-ai-lab-gui/ and pins it in .decimal/skills.lock

decimalai skills add svd-ai-lab-gui

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/svd-ai-lab-gui/SKILL.md https://app.decimal.ai/s/svd-ai-lab-gui/SKILL.md

Raw file: /s/svd-ai-lab-gui/SKILL.md ↗ · machine-readable summary at /s/svd-ai-lab-gui.json ↗

Safety checks: 3 of 3 passed

Code scan CleanIntent PassedContent Clean

See it in action

One of 19 test cases the skill fixed. Same model, same prompt, with and without the skill.

1 of 19

Prompt

In an active window dialog, you need to type the file path '/tmp/out.cas.h5' into the primary text box, but the text box lacks an accessible label name. How should you invoke send_text on the window object?

What was checked, without and with the skill
Checked	Without skill	With skill
Calls dlg.send_text("/tmp/out.cas.h5") omitting the into parameter.	Missed	Passed

case-13 · gemini-3.6-flash · 9/26/2026 · answers are capped and redacted

What's inside

The instructions your model reads when the skill loads.

SKILL.md233 linessingle file

`gui` — cross-driver GUI actuation

Whenever the active driver runs with ui_mode=gui (or desktop), sim serve injects a gui object into your sim exec namespace alongside session / solver / meshing / model. The object is the same shape across solvers — only the process filter differs — so one skill serves every GUI-capable driver.

/connect advertises it:

{
  "ok": true,
  "data": {
    "...": "...",
    "tools": ["gui"],
    "tool_refs": {"gui": "sim-cli/_skills/sim-cli/gui/SKILL.md"}
  }
}

If tools doesn't contain "gui", the driver launched headless and the object is absent — don't call it.

When to reach for `gui`

Three scenarios dominate:

A blocking dialog is wedging the workflow. A login prompt, overwrite confirmation, or script-error dialog can pause agent work until someone clicks a button — that someone is you, via gui.
You need to drive the GUI where the SDK can't. Some workflows expose a UI-only surface that the driver API does not cover.
You need a per-window screenshot. sim screenshot captures the whole desktop. SimWindow.screenshot() captures just the window you care about — cheaper to read, less visual clutter for the LLM.

If the SDK has a programmatic path (session.tui.*, model.solve(), ModelUtil.loadCopy()), prefer that. gui is for the UI-only surface that the SDK doesn't cover.

Track record

Benchmark

22 / 22

test cases passed, vs 3 / 22 without

Adoption

Users: 0
Times used: 0

Safety and source details

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v5 — 8/17/2026

View 15 scanner checks

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

svd-ai-lab / sim-cli

src/sim/_skills/sim-cli/gui/SKILL.md

View on GitHub ↗

License Apache-2.0Imported 8/17/2026Version v1

More skills from the same source and nearby in the registry, with their headline benchmark lift.