Validity Probing Strategy

Deep investigation of construct validity for individual benchmarks. Determines whether a benchmark actually measures the capability it claims to measure, or whether high scores can be achieved through shortcuts, artifacts, or unrelated competencies.

Purpose

Produce a construct validity assessment that identifies the gap between what a benchmark claims to measure and what it actually measures. Expose confounds, shortcuts, and alternative explanations for high performance.

Budget

| Resource | Floor | Target | |----------|-------|--------| | Benchmarks probed | 2 | 3 | | Papers read | 30 | 40 | | Web searches | 20 | 30 |

State Ledger

<HARD-GATE>
| Metric | Current | Target | Status |
|--------|---------|--------|--------|
| Benchmarks probed | 0 | 3 | PENDING |
| Papers fetched | 0 | 40 | PENDING |
| Papers read | 0 | 30 | PENDING |
| Web searches | 0 | 30 | PENDING |
| Construct validity assessments | 0 | 3 | PENDING |
| Artifact detection runs | 0 | 3 | PENDING |
| Alternative explanation catalogs | 0 | 3 | PENDING |
| Convergent validity checks | 0 | 3 | PENDING |
</HARD-GATE>

Cannot exit until 80% of all targets met.

Available Tactics

• artifact-detection — Detect annotation artifacts and shortcuts
• evaluation-protocol-comparison — Compare how different papers implement the benchmark

Available SOPs

• construct-validity-assessment — Core validity evaluation
• metric-decomposition — Understand what the metric actually captures
• contamination-audit — Rule out data leakage as confound
• benchmark-synthesis — Produce validity report

Execution Guidance

1. Target Selection: Choose 3 benchmarks where validity concerns exist (high scores but questionable real-world transfer, known shortcuts, or contested claims)
2. Per-Benchmark Deep Dive:

a. Collect the original benchmark paper + all critique/analysis papers b. Run construct-validity-assessment: map claimed capability to actual task requirements c. Run artifact-detection tactic: probe for shortcuts and spurious correlations d. Run metric-decomposition: identify what signals the metric actually rewards e. Check convergent validity: do models that score high also perform well on related tasks? f. Check discriminant validity: do models that score high fail on tasks they shouldn't if the capability were real?

1. Alternative Explanation Catalog: For each benchmark, enumerate non-target explanations for high scores
2. Synthesis: Produce validity verdict with evidence strength ratings

Output Format

yaml
validity_report:
  benchmark_name: string
  claimed_capability: string
  actual_measurement: string  # what it really measures
  validity_verdict: valid|partially_valid|questionable|invalid
  evidence_strength: strong|moderate|weak
  confounds_identified:
    - confound: string
      severity: high|medium|low
      evidence: string
  shortcuts_found:
    - shortcut: string
      exploit_method: string
      performance_gain: string
  convergent_validity: pass|partial|fail
  discriminant_validity: pass|partial|fail
  recommendations: list[string]

<!-- BEGIN available-tables (generated) -->

Available Tactics

Optional, no fixed order; the final leaf is always a sop.

| Tactic | When to use | | --- | --- | | artifact-detection | Detect annotation artifacts and shortcuts in benchmarks | | evaluation-protocol-comparison | Compare implementation differences of same benchmark across papers |

Available SOPs

Optional, no fixed order; the final leaf is always a sop.

| SOP | When to use | | --- | --- | | benchmark-synthesis | Produce final structured audit report | | construct-validity-assessment | Evaluate whether benchmark measures its claimed capability | | contamination-audit | Detect train-test data leakage and memorization artifacts | | metric-decomposition | Decompose composite metrics into constituent signals, analyze polarity and ceiling effects |

<!-- END available-tables (generated) -->