Measured 10/10/2026v3 · latest is v416 of 22 cases passed with skill
Tokens, turns and time describe resource use, not a dollar price.
This benchmark measured an earlier version of the skill.
A newer attempt on gemini-3.8-flash on 10/20/2026 did not produce a publishable comparison (not comparable). The snapshot above is the previously measured run.
See it in action
One of 8 test cases the skill fixed. Same model, same prompt, with and without the skill.
1 of 8
Prompt
Where does the security gate script store output summary JSON files by default, and what environment variable allows changing this target directory?
What was checked, without and with the skill
Checked
Without skill
With skill
Identifies SECURITY_GATE_OUTPUT_DIR as the override environment variable.
Passed
Passed
Identifies /tmp/agentops-security or TMPDIR as default base output path.
Missed
Passed
case-22 · gemini-3.6-flash · 10/10/2026 · answers are capped and redacted
What's inside
The instructions your model reads when the skill loads.
SKILL.md163 lines11 filesincludes scripts
Security Skill
Purpose: Find and report security weaknesses in code, scripts, authorized binaries, and repo-managed prompt surfaces, with honest coverage.
Use this skill for a caller-requested security review of code, a repository scan, authorized binary assurance, dependency risk, secrets, or offline prompt-surface redteam.
Critical Constraints
Scan only repositories, binaries, and prompt surfaces the operator owns or is explicitly authorized to assess. Why: a security review does not grant access to third-party systems or proprietary material.
Keep collection read-only by default; do not exfiltrate secrets, execute destructive payloads, or mutate policy/baselines to manufacture green. Why: the assessment must not become the incident or erase its evidence.
Treat missing/error scanners as a coverage gap, never a clean finding; use --require-tools when complete tool coverage is required. Why: absent evidence is not evidence of absence.
Use the current agent and local shell; do not start another runtime or orchestration substrate unless explicitly requested. Why: repository scanning is a bounded operation, not permission to fan out.
Report findings and coverage gaps, then stop. Remediation, risk acceptance,
reruns, promotion, and any ship or merge call are caller decisions. Name each
finding's remediation class in a few words; do not write the patch, a plan,
an owner, or a priority.
What every review reports
Apply these to every review, scripted or manual. They are the rules most often skipped:
Fail-open paths. For every guard, check, timeout, and exception handler
on the surface, ask what happens when it errors or hangs. A control that
grants access, skips a check, or continues as success on error is a finding
even when its happy path is correct.
Borrowed identity. Trace the effective identity at each hop (user,
service, token, default, hook). A hop where identity is assumed, defaulted,
or inherited instead of verified is the borrowed identity failure mode
and a finding.
Per-class coverage ledger. Walk every applicable class in
the OWASP checklist (the attack pack for
prompt surfaces), plus fail-open and identity, and give each a result:
finding, clean, or not assessed. An unvisited class is a gap, never a clean.
Chasing one lead to the exclusion of the taxonomy is the first-scent
fixation failure mode.
Proven versus suspected. A finding is proven only when you ran a
concrete input, request, or command and observed the behavior; capture it.
A finding reasoned from the code is suspected, even with a candidate input;
give that input and rank it below proven findings.