Passed 14 of 18 test cases with the skill, and 7 of 18 without it.
Without skill39%
With skill78%
Checks it passed that the model alone missed
Automatic retries and git commits are explicitly omitted.
The refactoring process stops without probing candidate seam C.
The transformation field names only the single change executed.
and 12 more in the full report
9 fixed5 passed either way2 still failing2 broke4 errored
Pass-rate lift
+39 pp
Tokens
+42%
Agent turns
0%
Response time
-34%
Model gemini-3.6-flashMeasured 10/10/2026v414 of 22 cases passed with skill · 18 comparable
Tokens, turns and time describe resource use, not a dollar price.
See it in action
One of 9 test cases the skill fixed. Same model, same prompt, with and without the skill.
1 of 9
Prompt
During a refactoring of `auth/session.go`, the post-refactor test check failed with 2 new errors. Should the refactoring tool automatically attempt to retry with a different approach, revert git commits, or escalate via ticket routing?
What was checked, without and with the skill
Checked
Without skill
With skill
The refactoring process stops after reporting the test failure evidence.
Passed
Passed
Automatic retries and git commits are explicitly omitted.
Missed
Passed
case-15 · gemini-3.6-flash · 10/10/2026 · answers are capped and redacted
What's inside
The instructions your model reads when the skill loads.
SKILL.md106 lines3 files
Refactor — one structural experiment
Refactor changes structure while preserving observable behavior. It performs one
caller-selected transformation and reports the result. "Behavior-preserving" is
a claim to prove with before/after checks, never to assert.
What counts as behavior
Unless the caller explicitly excluded a surface, all of these must survive:
Messages and exit codes. Error and output text compares byte-for-byte;
exit codes, error types and which input raises which error stay the same.
Scripts and callers parse them. Preserve an inconsistent message and report
it; normalizing it is a behavior change.
Differences between near-duplicates. When merging duplicated branches,
carry every difference (constants, comparisons, messages, extra steps) as a
parameter or a branch. Do not unify a difference the caller has not declared
accidental.
Interfaces. Public signatures, defaults, return types, persisted field
names, protocol values and CLI flags. Renaming one is a compatibility change
unless the accepted scope provides for it.
Order and coverage. Branch priority, default handling, evaluation count,
side-effect order, and the set of tests that run. A pre-existing red that
vanishes, or a test that stops running, is a behavior change.
Procedure
Name the preserved behavior, the focused acceptance surface and the concrete
structural problem for its callers, in the caller's domain terms.
Run the focused check and the smallest regression check the changed surface
justifies, and record that honest baseline, including reproducible ambient
failures. For an evaluation comparing executable behavior, pin the starting
source, build its baseline before edits and keep that binary and the
comparison inputs.
Apply one bounded transformation: extract, rename, inline, simplify,
encapsulate, move, or delete dead code. Judge it by what callers must
understand and where a domain rule must change, not by file size.
A bug or suspicious inconsistency found on the way is reported separately
(location, why it looks wrong) and left unfixed. Fixing it inside the
refactor hides a behavior change the caller did not authorize.