Evaluation Protocol Comparison Tactic

Compare how different papers implement the same benchmark to expose hidden protocol variance that undermines cross-paper score comparability.

Stages

Stage 1: Paper Collection (Same Benchmark)

Collect 10-15 papers that report results on the target benchmark:

• Prioritize diversity: different labs, years, model families
• Include the original benchmark paper as reference protocol
• Include papers from different venues (top conferences, workshops, preprints)
• Search via dare-ss (ss_relevance_search) and dare-scholar (paper_searching)

Search queries: "benchmark name] evaluation", "benchmark name] results", "benchmark name] state-of-the-art"

Stage 2: Protocol Element Extraction

For each paper, run protocol-element-extraction SOP to extract:

| Element Category | Specific Parameters | |-----------------|-------------------| | Data | Split version, subset selection, preprocessing, filtering | | Prompting | Template format, few-shot examples (count, selection), instruction wording | | Generation | Decoding strategy, temperature, top-p/top-k, max tokens, stop criteria | | Evaluation | Metric implementation, postprocessing, normalization, scoring script version | | Infrastructure | Framework, precision (fp16/bf16/fp32), batch size, hardware |

Stage 3: Difference Matrix Construction

Build a comparison matrix:

• Rows = protocol elements
• Columns = papers
• Cells = specific value used
• Highlight deviations from original protocol

Compute per-element variance:

• None: All papers use identical value
• Low: Minor variations (e.g., different random seeds)
• Medium: Substantive differences (e.g., different few-shot examples)
• High: Fundamental disagreements (e.g., different splits, different metrics)
• Extreme: Papers appear to evaluate different things under same name

Stage 4: Impact Assessment

For each high-variance element:

• Search for ablation studies showing impact of that element
• Estimate score range attributable to protocol choice vs model quality
• Identify which protocol choices systematically favor certain model families
• Flag "protocol p-hacking" — suspicious correlation between protocol choice and reported improvement

Output

yaml
protocol_comparison:
  benchmark: string
  papers_compared: int
  reference_protocol: string  # original benchmark paper
  difference_matrix:
    - element: string
      category: data|prompting|generation|evaluation|infrastructure
      variance_level: none|low|medium|high|extreme
      values: list[{paper, value}]
      impact_estimate: string
  highest_variance_elements:
    - element: string
      score_impact: string
      favors: string  # which model family benefits
  protocol_p_hacking_flags:
    - paper: string
      suspicious_choice: string
      benefit: string
  cross_paper_comparability: high|moderate|low|unreliable
  standardization_recommendations:
    - element: string
      recommended_value: string
      rationale: string

Yield Report

| Metric | Minimum | |--------|---------| | Papers compared | 8 | | Protocol elements extracted per paper | 10 | | High-variance elements identified | 2 | | Impact estimates produced | 3 |

<!-- BEGIN available-tables (generated) -->

Available SOPs

Optional, no fixed order; the final leaf is always a sop.

| SOP | When to use | | --- | --- | | protocol-element-extraction | Extract evaluation protocol parameters from papers |

<!-- END available-tables (generated) -->