Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

Get Started Free →

E2e · vellum-ai | DecimalAI

testingby vellum-ai

E2e

vellum-ai/e2e

Source vellum-ai/vellum-assistant ↗License MIT

Run end-to-end tests via the CI workflow.

SkillScore18Maintained

Automatic

QA & Test Engineering pack

Last verified 8/4/2026

What it changes

Passed 1 of 8 test cases with the skill, and 1 of 8 without it.

Without skill14%

With skill18%

Checks it passed that the model alone missed

Includes -d flag
Includes --xcode flag
Passes -b release/1.2

and 2 more in the full report

3 fixed1 passed either way16 still failing2 broke

Pass-rate lift: +5 pp
Tokens: +15%
Agent turns: 0%
Response time: -31%

Model gemini-3.6-flashMeasured 8/4/2026v14 of 22 cases passed with skill · 8 comparable

Tokens, turns and time describe resource use, not a dollar price.

Use this skill

Save it into your project's skills folder. Claude Code picks it up in its next session.

curl -fsSL --create-dirs -o .claude/skills/vellum-ai-e2e/SKILL.md https://app.decimal.ai/s/vellum-ai-e2e/SKILL.md

Want it pinned and updatable? pip install "decimalai>=0.15", then decimalai skills add vellum-ai-e2e, and start Claude Code with claude --plugin-dir "$(git rev-parse --show-toplevel)/.agents"

1 · One-time — install the CLI (docs ↗)

pip install "decimalai>=0.15"

2 · Get the skill — writes .agents/skills/vellum-ai-e2e/ and pins it in .decimal/skills.lock

decimalai skills add vellum-ai-e2e

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/vellum-ai-e2e/SKILL.md https://app.decimal.ai/s/vellum-ai-e2e/SKILL.md

Raw file: /s/vellum-ai-e2e/SKILL.md ↗ · machine-readable summary at /s/vellum-ai-e2e.json ↗

Safety checks: 3 of 3 passed

Code scan CleanIntent PassedContent Clean

See it in action

One of 3 test cases the skill fixed. Same model, same prompt, with and without the skill.

1 of 3

Prompt

Run e2e test case onboarding-flow using the agent-xcode runner in detached mode on feature branch feat/onboard.

What was checked, without and with the skill
Checked	Without skill	With skill
Includes --xcode flag	Missed	Passed
Includes -d flag	Missed	Passed

case-15 · gemini-3.6-flash · 8/4/2026 · answers are capped and redacted

What's inside

The instructions your model reads when the skill loads.

SKILL.md50 linessingle file

Run end-to-end tests via the CI workflow.

The user may pass $ARGUMENTS to filter to a specific test case by name (e.g., hello-world, phone-setup). If not provided, infer options from context (see below).

Steps

1. Determine options

Experimental tests: Always include --experimental by default. The user can pass --no-experimental to exclude them.

Branch detection: Check what branch you're on via git branch --show-current. If you're on a branch other than main, automatically pass -b <branch-name> so the CI run tests against that branch. The user can override with -b <other-branch> explicitly.

Test case filter: Determine which test case to target:

If the user passes a bare word argument (e.g., phone-setup), use that as the filter.
Otherwise, look at the conversational context - if you've been working on or discussing a specific e2e test case (e.g., editing playwright/cases/phone-setup.md), automatically target that test case.
If neither applies, run all tests.

Additional flags from $ARGUMENTS:

--xcode uses the agent-xcode (AXUIElement) runner
-d or --detach triggers the run and exits without polling
--no-experimental overrides the default and excludes experimental tests

2. Trigger the CI run

cd playwright && bun run scripts/agent-ci.ts <options>

Map the resolved options:

Test case filter: -t <case-name>
Experimental (default on): --experimental
Branch: -b <branch-name>
Xcode runner: --xcode
Detach mode: -d

Examples:

/e2e (on main, no context) → bun run scripts/agent-ci.ts --experimental
/e2e (on branch feat/phone, after editing phone-setup.md) → bun run scripts/agent-ci.ts --experimental -b feat/phone -t phone-setup
/e2e hello-world (on main) → bun run scripts/agent-ci.ts --experimental -t hello-world
/e2e --detach (on branch fix/bug) → bun run scripts/agent-ci.ts --experimental -b fix/bug -d

Track record

Benchmark

1 / 8

test cases passed, vs 1 / 8 without

Adoption

Users: 0
Times used: 0

Safety and source details

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v4 — 8/1/2026

View 15 scanner checks

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

vellum-ai / vellum-assistant

.claude/skills/vellum-skills/e2e/SKILL.md

View on GitHub ↗

License MITImported 8/1/2026Version v1

More skills from the same source and nearby in the registry, with their headline benchmark lift.