Test Suite Auditor

Test Suite Auditor

Goal: Audit the test portfolio as a read-only lifecycle and confidence system. Determine which important failures it detects, which evidence is untrustworthy or obsolete, and which additions, changes, consolidations, retirements, or explicit omissions produce the smallest sustainable portfolio.

Execution contract: The ordered checkboxes are the Definition of Done. Track every item internally as PENDING, PROVEN with concrete evidence, CLEARED with evidence that its condition is absent, or UNPROVEN with a gap; reading, delegation, or tool failure is not proof. Reconcile items after each section. Before returning, resolve all PENDING and count only PROVEN and CLEARED; apply the skill's verdict and approval rules to every gap. Preserve user intent, scope, and existing authorization. Continue authorized work; ask only for consequential unresolved choices or required external approval. Scale depth to material risk without silently skipping checks. Preserve dependency and safety ordering; otherwise choose the verification method appropriate to each obligation.

Tool Routing

Run only safe test and diagnostic commands. Do not rewrite snapshots, update golden files, regenerate fixtures, or accept changed output during the audit.

Evidence Rules

• Coverage indicates execution, not proof. Require an assertion or observable oracle for important behavior.
• A slow test is not low-value when it uniquely protects a critical journey; a fast test is not high-value when it proves framework behavior.
• A flaky failure must be separated from an intermittently failing product dependency or genuinely nondeterministic requirement.
• Deletion recommendations require proof that the test basis is obsolete or that other evidence covers every still-required behavior and failure mode with equal or better trust.
• Merge recommendations require demonstrated duplicate or fragmented coverage and must preserve distinct business and failure scenarios, oracle strength, and failure localization; a larger test is not inherently better.
• Known regression guards and the only proof of a rare critical edge case are not deletion candidates merely because a numeric value heuristic is low.
• A real dependency is not inherently a test defect. Judge whether its version, state, ownership, reset, availability, and failure evidence make the result reproducible.
• Keep portfolio action (KEEP, ADD, UPDATE, MERGE, DELETE, NO_TEST) separate from execution status (PASS, FAIL, BLOCKED, UNPROVEN, QUARANTINED). NO_TEST requires existing proof, another control, or explicit residual-risk acceptance.
• Do not require a new registry by default. Prefer traceability derivable from repository-native test paths, behavioral names, tags, requirements, CI configuration, and review evidence.
• External testing guidance becomes actionable only when it explains a concrete weakness in this suite.

Checklist

1. Map the Portfolio and Baseline

• Detect test runners, configurations, commands, directories, fixtures, factories, snapshots, golden files, manual scripts, coverage, and mutation tooling.
• Map source domains and critical entrypoints to unit, integration, contract, end-to-end, and manual test surfaces.