Passed 5 of 12 test cases with the skill, and 3 of 12 without it.
Without skill22%
With skill39%
Checks it passed that the model alone missed
Executes pnpm dedupe following the package upgrade.
Executes pnpm why -r semver as the final verification step.
Stops execution by treating the request as a major migration.
and 7 more in the full report
6 fixed3 passed either way12 still failing2 broke
Pass-rate lift
+17 pp
Tokens
+21%
Agent turns
0%
Response time
-35%
Model gemini-3.6-flashMeasured 9/27/2026v29 of 23 cases passed with skill · 12 comparable
Tokens, turns and time describe resource use, not a dollar price.
See it in action
One of 6 test cases the skill fixed. Same model, same prompt, with and without the skill.
1 of 6
Prompt
We are upgrading @clerk/nextjs to version 5.1.0. What exact git command should be provided in the final output for branching and committing?
What was checked, without and with the skill
Checked
Without skill
With skill
Uses clerk-nextjs as the branch name slug in git switch -C deps/bump-clerk-nextjs-to-5.1.0.
Missed
Passed
Preserves @clerk/nextjs in git commit -m "chore(deps): bump @clerk/nextjs to 5.1.0" --no-verify.
Missed
Passed
case-15 · gemini-3.6-flash · 9/27/2026 · answers are capped and redacted
What's inside
The instructions your model reads when the skill loads.
SKILL.md89 lines2 filesincludes scripts
PNPM Upgrade Package
Use this skill for interactive dependency bumps in Langfuse.
Read Order
Use this SKILL.md for the end-to-end workflow.
Run the main helper once at the start of the upgrade:
node .agents/skills/pnpm-upgrade-package/scripts/check-release-age-window.mjs <package> [targetVersion]
Apply This Skill
Ask for the package name if the user did not provide one.
Ask for the target version if the user did not provide one.
Run the main helper once as the first analysis step and use that single
output for scope, exclusion decisions, and the final bump.
If the target package is not directly declared anywhere, run
pnpm why -r <package> to find which direct dependency brings it in, then
inspect whether the current top-level parent already allows the requested
transitive version via its dependency range.
If the current parent range already covers the requested transitive version,
prefer a lockfile refresh / reinstall path over bumping the parent manifest.
If the current parent range does not cover the requested transitive version,
upgrade that parent dependency instead of adding the target package directly
unless the user explicitly wants that.
Probe the parent once at @latest:
npm view <parent>@latest dependencies peerDependencies optionalDependencies --json.
If that range resolves to any non-vulnerable version of the target, even one
newer than the lowest fix, upgrade the parent instead of pinning the exact
lowest fix. If the latest parent still pins a vulnerable range, do not walk
intermediate parent versions: add a scoped overrides entry in
pnpm-workspace.yaml only when the fixed version stays within the major the
parent declares, otherwise treat it as a major migration and stop. In the
final response, state that the parent pins the vulnerable range so the
reviewer knows the override is a workaround.
If pnpm will not move an already-allowed transitive version, a scoped
overrides entry in pnpm-workspace.yaml may be used as a temporary
resolution tool. Before finishing, prove whether the override is still
required: remove it, run pnpm install, then run pnpm dedupe. Inspect the
diff after each generated change. If the target version remains without the