Run an A/B codex review experiment — holistic codex review vs 3 focused dimension passes (security, ecto, liveview) on the branch diff, classify findings, report a panel-value verdict.
SkillScore77Maintained
Automatic
◆Coding pack
Last verified 8/19/2026
What it changes
Passed 15 of 19 test cases with the skill, and 3 of 19 without it.
Without skill14%
With skill77%
Checks it passed that the model alone missed
Specifies running command -v codex.
Classifies the finding as DUPLICATE.
Classifies the finding as REAL MISS.
and 15 more in the full report
15 fixed2 passed either way4 still failing1 broke
Pass-rate lift
+64 pp
Tokens
+3%
Agent turns
0%
Response time
-54%
Model gemini-3.6-flashMeasured 8/19/2026v117 of 22 cases passed with skill · 19 comparable
Tokens, turns and time describe resource use, not a dollar price.
See it in action
One of 14 test cases the skill fixed. Same model, same prompt, with and without the skill.
1 of 14
Prompt
The security worker reported an unescaped raw HTML snippet on line 88 of template.html.heex. Holistic review did not mention line 88. After opening template.html.heex, we verified that line 88 contains raw dynamic input without sanitization introduced in this branch. How is this finding classified?
What was checked, without and with the skill
Checked
Without skill
With skill
Classifies the finding as REAL MISS.
Missed
Passed
case-11 · gemini-3.6-flash · 8/19/2026 · answers are capped and redacted
What's inside
The instructions your model reads when the skill loads.
Answer one question with evidence: do dimension-focused codex passes find
real issues that one holistic codex exec review misses? Runs both on the
same diff, then classifies every focused finding against the holistic pass.
DECIDED 2026-07-10 after 4 runs (2 fresh): panel KILLED. Fresh-only
1 real miss / 1 false positive plus one zero-value run at 4× cost —
real misses did not outnumber FPs. Kept as contributor tooling (NOT
distributed) for one possible retest: a UI-heavy diff with a single
extra liveview-focused pass (2× cost). Scoreboard:
.claude/research/2026-07-03-codex-review-integration.md §7.
Usage
/codex-ab # A/B against main (~5 min, 4 codex runs)
/codex-ab develop # explicit base branch
Iron Laws
FRESH DIFF ONLY — ask the user to confirm this branch has NOT been
codex-reviewed yet (cloud or /phx:codex-loop). A drained diff returns
NO FINDINGS everywhere and proves nothing — wasted quota
Verify every REAL MISS in the code before counting it — a focused
finding only scores if the issue actually exists at that file:line
Read ONLY the findings .md files — streams are diverted to .log
files; never cat a log into context (10k+ lines each)
Exactly 4 codex runs, never re-run dimensions — bounded quota
Persist the verdict — an unrecorded experiment is wasted quota
Workflow
Step 1: Preflight
Run command -v codex — missing → STOP with install hint. Then:
git status --short dirty → warn (codex flags local dirt as findings)
Ask: "Has codex already reviewed this branch (PR review or codex-loop)?"
Bump outdated Hex deps — inventory, snapshot changelogs, update, fix breaks, split reviewable PRs (patches bundled, majors solo). Use to upgrade/bump Elixir dependencies or when versions fall behind. NOT for deps.get failures (/phx:investigate).