Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

run-goldens — run the golden LLM tests against a real model, by choice · fedoroff-vlad | DecimalAI

testingby fedoroff-vlad

run-goldens — run the golden LLM tests against a real model, by choice

fedoroff-vlad/run-goldens

Source fedoroff-vlad/agent-skills ↗License MIT

Use when you want to run the golden LLM tests against a real model — one, a few, or all. Fires on: "прогони голдены" / "run the goldens", "прогони golden тесты", "проверь роутинг/синтез на реальной модели", "did my prompt/skill/router change break the goldens", after editing a prompt, SKILL.md, classifier, or any LLM surface. Discovers the repo's golden tests, lets you pick a scope, runs them through the repo's golden runner, and reads the result (real regression vs flaky borderline case).

SkillScore86Maintained

Automatic

QA & Test Engineering pack

Last verified 8/6/2026

What it changes

Passed 18 of 21 test cases with the skill, and 7 of 21 without it.

Without skill32%

With skill86%

Checks it passed that the model alone missed

The response states to re-run the failing test case exactly once.
The response mentions that the runner sets GOLDEN_LLM=true automatically.
The response states that golden test inputs must be crisp and unambiguous.

and 10 more in the full report

12 fixed7 passed either way3 still failing0 broke

Pass-rate lift: +55 pp
Tokens: +10%
Agent turns: 0%
Response time: -64%

Model gemini-3.6-flashMeasured 8/6/2026v1 · latest is v219 of 22 cases passed with skill · 21 comparable

Tokens, turns and time describe resource use, not a dollar price.

This benchmark measured an earlier version of the skill.

A newer attempt on gemini-3.6-flash on 10/2/2026 did not produce a publishable comparison (not comparable). The snapshot above is the previously measured run.

Use this skill

Save it into your project's skills folder. Claude Code picks it up in its next session.

curl -fsSL --create-dirs -o .claude/skills/fedoroff-vlad-run-goldens/SKILL.md https://app.decimal.ai/s/fedoroff-vlad-run-goldens/SKILL.md

Want it pinned and updatable? pip install "decimalai>=0.15", then decimalai skills add fedoroff-vlad-run-goldens, and start Claude Code with claude --plugin-dir "$(git rev-parse --show-toplevel)/.agents"

1 · One-time — install the CLI (docs ↗)

pip install "decimalai>=0.15"

2 · Get the skill — writes .agents/skills/fedoroff-vlad-run-goldens/ and pins it in .decimal/skills.lock

decimalai skills add fedoroff-vlad-run-goldens

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/fedoroff-vlad-run-goldens/SKILL.md https://app.decimal.ai/s/fedoroff-vlad-run-goldens/SKILL.md

Raw file: /s/fedoroff-vlad-run-goldens/SKILL.md ↗ · machine-readable summary at /s/fedoroff-vlad-run-goldens.json ↗

Safety checks: 3 of 3 passed

Code scan CleanIntent PassedContent Clean

See it in action

One of 12 test cases the skill fixed. Same model, same prompt, with and without the skill.

1 of 12

Prompt

I only want to execute the test method `testFinanceRouting` inside the class `RoutingGoldenTest` in the `router-service` module using the repository runner. What is the exact command line invocation?

What was checked, without and with the skill
Checked	Without skill	With skill
The response uses scripts/golden.sh with -pl router-service and -Dtest=RoutingGoldenTest#testFinanceRouting.	Missed	Passed

case-06 · gemini-3.6-flash · 8/6/2026 · answers are capped and redacted

What's inside

The instructions your model reads when the skill loads.

SKILL.md95 linessingle file

run-goldens — run the golden LLM tests against a real model, by choice

Golden tests (@GoldenLlmTest-style) exercise an LLM surface — routing, strict-JSON extraction, synthesis grounding — against a real local model, asserting structure, not exact text. They are opt-in (a GOLDEN_LLM gate), skipped in CI, and slow (a real model call per assertion), so you run the ones that touch what you changed — not the whole suite reflexively.

Procedure

Discover the goldens. Find the repo's golden classes — grep for the golden marker (this repo family uses @GoldenLlmTest): grep -rl "@GoldenLlmTest" --include=*.java (adapt the marker/glob to the repo's language). Group the hits by module (the Maven -pl path / package root) so each has a runnable address.
Present the inventory + get a scope. Show the discovered classes grouped by module and ask which to run: one class, several (comma-separated), a whole module, or all. You can narrow to a single method with Class#method. Prefer the smallest scope that covers the surface just changed.
Check prerequisites. The golden lane needs a real model. Use the repo's golden runner (scripts/golden.sh in this repo family) — it auto-starts the local inference engine + an LLM gateway pointed at it, sets GOLDEN_LLM=true, and handles model quirks (e.g. suppressing "thinking" so a call is seconds not minutes). The required model must already be pulled; the runner names the pull command rather than downloading multi-GB blobs unasked. Do NOT hand-roll the env — go through the runner so the setup stays identical to everyone else's.
Run the chosen scope through the runner, e.g.:
- one class: scripts/golden.sh -pl <module> -Dtest=<GoldenClass>
- several: scripts/golden.sh -pl <module> -Dtest='<GoldenA>,<GoldenB>'
- one method: scripts/golden.sh -pl <module> -Dtest=<GoldenClass>#<method>
- all: run per module (loop the -pl <module> -Dtest=<its Golden*> addresses). The runner leaves the engine + gateway up, so the next run is instant.
Read the result — real regression vs flaky case. On failure, extract only the failing assertion (e.g. «…» should route to 'finance' but went to 'tasks'), not the full log. Then apply the flakiness rule below before calling it a bug.
Report. A short per-class pass/fail; for each failure, the assertion + a verdict: real regression (stable, reproducible) or flaky borderline case (input too ambiguous for the model — tighten the test, not the product).

Flakiness — the rule that keeps goldens trustworthy

Track record

Benchmark

18 / 21

test cases passed, vs 7 / 21 without

Adoption

Users: 0
Times used: 0

Safety and source details

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v5 — 9/30/2026

View 15 scanner checks

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

fedoroff-vlad / agent-skills

skills/run-goldens/SKILL.md

View on GitHub ↗

License MITImported 9/30/2026Version v2

More skills from the same source and nearby in the registry, with their headline benchmark lift.

Ready to supercharge your AI agents?

See it in action

What's inside

run-goldens — run the golden LLM tests against a real model, by choice

Procedure

Flakiness — the rule that keeps goldens trustworthy

Track record

Benchmark

Adoption

SkillSafety

Source

Related skills