Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

Get Started Free →

Protocol Forensics Strategy · yogsoth-ai | DecimalAI

yogsoth-ai/protocol-forensics

SkillScore38Maintained

Analyze evaluation protocol differences across papers for same benchmark — 5 benchmarks, 60 papers, 30 web searches

researchby yogsoth-ai

Links this skill to your agents and keeps it current with the author’s updates.

Automatic

Research pack

1 · One-time — install the CLI (docs ↗)

pip install decimalai

2 · Get the skill — writes .claude/skills/yogsoth-ai-protocol-forensics/SKILL.md

decimalai skills pull yogsoth-ai-protocol-forensics --out .claude/skills/

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/yogsoth-ai-protocol-forensics/SKILL.md https://app.decimal.ai/s/yogsoth-ai-protocol-forensics/SKILL.md

Raw file: /s/yogsoth-ai-protocol-forensics/SKILL.md ↗ · machine-readable summary at /s/yogsoth-ai-protocol-forensics.json ↗

In productionGlobal

how this skill behaves on real traffic · last 30 days

No production data for this skill yet — usage numbers appear here once agents start running it.

Anonymized aggregate across every org using this skill — proof from real production use, not a demo.

Does this skill help? · Benchmark: with vs. withoutsign in to run this on your agents

24 test cases from the publisher’s eval.yaml · each run twice on gemini-3.6-flash — with the skill injected, then without · last run 17d ago

—+41% tokens0% turns38% of whole cases pass with skill (9/24)

Test case	Without → With	Effect	Δ tokens	Δ turns
case-09	✗→✓	▲ Improved	12%	0%
case-13	✗→✓	▲ Improved	14%	0%
case-14	✗→✓	▲ Improved	55%	0%
case-16	✗→✓	▲ Improved	190%	0%
case-21	✗→✓	▲ Improved	12%	0%

Showing 5 of 24 cases

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v4 — 8/10/2026

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

yogsoth-ai / de-anthropocentric-research-engine

skills/protocol-forensics/SKILL.md

View on GitHub ↗

License Apache-2.0Imported 8/10/2026Version v1

Skill contents

SKILL.md

Protocol Forensics Strategy

Forensic analysis of how the same benchmark is implemented differently across papers. Reveals hidden variance in evaluation protocols that makes cross-paper score comparisons unreliable.

Purpose

Expose the "reproducibility gap" in benchmark evaluation by documenting how papers differ in their implementation of supposedly standardized evaluation protocols. Quantify the score variance attributable to protocol differences rather than model improvements.

Budget

| Resource | Floor | Target | |----------|-------|--------| | Benchmarks forensically analyzed | 3 | 5 | | Papers read | 45 | 60 | | Web searches | 20 | 30 |

State Ledger

<HARD-GATE>
| Metric | Current | Target | Status |
|--------|---------|--------|--------|
| Benchmarks analyzed | 0 | 5 | PENDING |
| Papers fetched | 0 | 60 | PENDING |
| Papers read | 0 | 45 | PENDING |
| Web searches | 0 | 30 | PENDING |
| Protocol extractions complete | 0 | 60 | PENDING |
| Difference matrices built | 0 | 5 | PENDING |
| Variance attributions done | 0 | 5 | PENDING |
| Impact assessments complete | 0 | 5 | PENDING |
</HARD-GATE>

Cannot exit until 80% of all targets met.

Available Tactics

evaluation-protocol-comparison — Core tactic: compare protocol implementations across papers

Available SOPs

benchmark-inventory — Identify target benchmarks
protocol-element-extraction — Extract protocol parameters from individual papers
benchmark-synthesis — Produce forensics report

Execution Guidance

Target Selection: Choose 5 benchmarks with known reproducibility issues or high paper volume
Paper Collection (per benchmark):

a. Collect 10-15 papers that report results on the same benchmark b. Prioritize papers from different labs, time periods, and model families

Protocol Extraction (per paper):

a. Run protocol-element-extraction on each paper b. Extract: prompt format, few-shot examples, decoding parameters, evaluation script version, data split, preprocessing

Comparison (per benchmark):

a. Run evaluation-protocol-comparison tactic to build difference matrix b. Identify which protocol elements vary most across papers c. Estimate score impact of each protocol difference

Variance Attribution: Decompose reported score improvements into:

Genuine model improvement
Protocol optimization (prompt engineering, example selection)
Implementation differences (tokenization, postprocessing)

Synthesis: Produce per-benchmark forensics report with reproducibility recommendations

Output Format

yaml
protocol_forensics:
  benchmark_name: string
  papers_analyzed: int
  protocol_elements:
    - element: string  # e.g., "few-shot examples", "decoding temperature"
      variation_level: none|low|medium|high|extreme
      values_observed: list[string]
      score_impact_estimate: string
  variance_attribution:
    genuine_improvement: float  # proportion of score gains
    protocol_optimization: float
    implementation_differences: float
    unexplained: float
  most_impactful_differences:
    - element: string
      score_range: string  # e.g., "+3.2 to +7.8 points"
      evidence: string
  reproducibility_grade: A|B|C|D|F
  recommendations:
    - recommendation: string
      priority: high|medium|low

Available Tactics

Optional, no fixed order; the final leaf is always a sop.

| Tactic | When to use | | --- | --- | | evaluation-protocol-comparison | Compare implementation differences of same benchmark across papers |

Available SOPs

Optional, no fixed order; the final leaf is always a sop.

| SOP | When to use | | --- | --- | | benchmark-synthesis | Produce final structured audit report | | knowledge-acquisition-benchmark-inventory | Identify and catalog all relevant benchmarks in target domain | | protocol-element-extraction | Extract evaluation protocol parameters from papers |

Illustrative Want to see what yogsoth-ai/protocol-forensics would put in front of your model? Preview the rendered body and check its trigger phrases — no LLM call, no signup.

Latest benchmark · publisher runVerified

9/4/2026, 04:55 UTC · 24 cases

Measured on 16 comparable cases — below the 22 needed to publish a lift, so this result is shown here but not on the registry card or leaderboard.

Lift

+25%

38% vs 13%

Agent turns

0%

1.0 vs 1.0

Tokens

+41%

8,123.69 vs 5,779.19

fail → pass · 6pass → pass · 3fail → fail · 15

Per-case results (24)

Case	Status	Duration (ms)			Turns			Tokens			Tool calls
Case	Status	Without	With	Δ	Without	With	Δ	Without	With	Δ	Without	With	Δ
case-01 We are observing significant discrepancies in reported scores for the HumanEval benchmark across different recent publications. Please conduct a forensic protocol audit across these publications to isolate how evaluation setups differ. Output a structured YAML block under `protocol_forensics` detailing the benchmark name, count of publications reviewed, each extracted evaluation element with observed variations and impact estimates, a quantitative breakdown decomposing score variance into genuine model gains, protocol tuning, implementation nuances, and unexplained factors, followed by the most impactful differences, an overall letter grade for reproducibility, and actionable recommendations sorted by priority.	fail→fail	26,173	17,241	-34%	1	1	0%	4,049	1,627	-60%	0	0	—
case-02 Our team is investigating whether claimed benchmark improvements on MATH are driven by architectural gains or hidden differences in evaluation methodology across labs. Run a protocol forensics investigation on MATH papers to dissect prompt formats, shot counts, decoding parameters, and scoring scripts. Return the analysis as YAML structured with the benchmark's title, number of papers audited, detailed protocol elements showing observed variation levels and impact estimates, score variance attribution split into real model progress, prompt optimization, and technical differences, top protocol variations with point ranges and evidence, a letter grade evaluating reproducibility, and prioritized recommendations to fix protocol drift.	fail→fail	28,620	25,379	-11%	1	1	0%	4,754	2,582	-46%	0	0	—
case-19 TriviaQA open-domain QA evaluations differ across papers in whether answer normalization (lowercasing, punctuation stripping, article removal) is applied before or after regex matching. Conduct a forensic protocol comparison across TriviaQA literature and produce the YAML output.	fail→fail	41,465	59,891	+44%	1	1	0%	5,071	9,215	+82%	0	0	—
case-20 DocVQA document visual question answering papers show evaluation protocol drift in OCR text extractions, prompt bounding box inclusion, and ANLS metric thresholding. Perform a protocol forensics audit of 15 DocVQA papers and return structured YAML.	fail→fail	48,564	48,873	+1%	1	1	0%	8,237	9,215	+12%	0	0	—
case-03 I suspect cross-paper score comparisons for the MMLU dataset are subject to protocol variations like few-shot prompt construction and answer parser differences. Perform a forensic protocol audit across MMLU papers from multiple research organizations. Deliver your findings formatted in YAML with `protocol_forensics` as the top key, containing benchmark details, paper sample size, an array of protocol elements with variation severity and score impact, the percentage decomposition of score gains across genuine architecture progress versus evaluation tuning versus code differences, the specific protocol discrepancies that cause the largest score swings with supporting evidence, an overall letter grade (A-F), and prioritized steps to standardize future evaluations.	fail→fail	35,030	52,247	+49%	1	1	0%	6,342	9,297	+47%	0	0	—
case-04 We suspect reported GSM8K math reasoning performance varies significantly depending on whether papers use 8-shot standard prompts, chain-of-thought prompting, or custom regex answer extractors. Perform a forensic protocol comparison across 50 papers evaluating LLMs on GSM8K. Produce YAML output summarizing the forensic investigation.	fail→fail	48,091	20,034	-58%	1	1	0%	8,252	2,053	-75%	0	0	—
case-05 Repository-level software engineering benchmarks like SWE-bench suffer from evaluation protocol drift across papers due to differing test harness environments, container timeouts, and patch applying scripts. Conduct a protocol forensics audit of 15 papers evaluating on SWE-bench and present the results in YAML.	fail→fail	49,092	54,092	+10%	1	1	0%	8,243	9,221	+12%	0	0	—
case-06 Multiple papers reporting HellaSwag accuracy disagree on whether 0-shot, 10-shot, or 20-shot evaluation was conducted, leading to misleading progress claims. Conduct a forensic analysis of HellaSwag evaluation implementations across 40 papers and present the findings as structured YAML.	fail→fail	43,663	21,433	-51%	1	1	0%	8,251	1,764	-79%	0	0	—
case-07 TruthfulQA benchmarks are scored using automated judges, MC1/MC2 multi-choice framing, or generative BLEU/ROUGE metrics. Run a protocol forensics audit on TruthfulQA across 30 published papers to quantify how much score drift stems from judge model choice versus model honesty improvements. Provide a structured YAML report.	fail→fail	49,597	56,969	+15%	1	1	0%	8,256	9,234	+12%	0	0	—
case-08 AI2 Reasoning Challenge (ARC-Challenge) paper results show wide variance. Some labs use choices formatted as (A,B,C,D) while others use (1,2,3,4), and some adjust log-likelihood normalization across choices. Perform a forensic protocol audit across ARC-Challenge papers and output YAML under `protocol_forensics`.	fail→fail	33,360	51,765	+55%	1	1	0%	5,766	9,239	+60%	0	0	—
case-09 HumanEval+ evaluates LLM code generation using extended test suites, but papers differ in prompt formatting, docstring inclusion, and execution timeout settings. Conduct a protocol forensics analysis across 20 papers on HumanEval+ and return the YAML output.	fail→pass	39,467	48,534	+23%	1	1	0%	8,238	9,216	+12%	0	0	—
case-10 AlpacaEval results vary depending on whether GPT-4-Turbo, Claude 3, or Llama 3 is used as the evaluator judge, as well as prompt length normalization. Run a forensic protocol comparison across papers reporting AlpacaEval win rates and format the output in YAML.	fail→fail	37,808	101,853	+169%	1	1	0%	5,841	9,225	+58%	0	0	—
case-11 BIG-bench Hard (BBH) spans 23 algorithmic tasks, but publications differ on whether 3-shot or 5-shot CoT is used, and whether string matching or regex is applied. Conduct a forensic protocol audit across BBH literature and return the structured YAML report.	fail→fail	40,242	14,280	-65%	1	1	0%	8,248	1,707	-79%	0	0	—
case-12 MBPP python code synthesis evaluations show discrepancies due to assertion checking logic, prompt function signature insertion, and pass@k sample sizes. Audit 25 papers using MBPP for protocol variations and return the forensic report in YAML.	fail→fail	47,445	61,956	+31%	1	1	0%	8,234	9,212	+12%	0	0	—
case-13 When auditing MMLU protocol differences across research papers, our team wants to classify the degree of divergence in decoding temperature settings (0.0 vs 0.7) and token probability scoring methods. Produce a YAML report for MMLU under `protocol_forensics` where variation levels are categorized.	fail→pass	27,816	24,307	-13%	1	1	0%	4,008	4,576	+14%	0	0	—
case-14 A protocol forensics audit on the MATH dataset reveals severe inconsistencies in LaTeX parsing scripts and sympy evaluation libraries across 35 AI research publications. Return the audit findings as YAML under `protocol_forensics` with a letter grade assessing overall benchmark reproducibility.	fail→pass	19,293	19,484	+1%	1	1	0%	2,291	3,551	+55%	0	0	—
case-15 After auditing SWE-bench evaluation harness differences across recent paper submissions, generate actionable recommendations to fix evaluation drift in future benchmarks. Output the results in YAML as part of the `protocol_forensics` summary structure.	fail→fail	19,516	18,958	-3%	1	1	0%	2,332	1,858	-20%	0	0	—
case-16 In our forensic audit of agentic coding benchmark papers, we want to isolate how much of the reported performance gains come from real model architecture progress versus prompt tuning or evaluation script patches. Generate the YAML audit report with variance attribution breakdown.	fail→pass	24,318	48,476	+99%	1	1	0%	3,180	9,213	+190%	0	0	—
case-17 We reviewed 30 papers evaluating open-weights LLMs on GSM8K and noticed huge score jumps caused solely by switching from exact match string comparison to flexible numerical parsing. Perform a forensic protocol audit and output the variance decomposition as YAML.	pass→pass	24,486	58,947	+141%	1	1	0%	3,772	8,686	+130%	0	0	—
case-18 AGIEval combines standardized admission exams like SAT, GRE, and LSAT, but papers differ in prompt translation, multi-choice formatting, and score aggregation across subsets. Perform a forensic protocol investigation across 20 AGIEval papers and present the output in YAML format.	fail→fail	51,329	22,466	-56%	1	1	0%	8,244	1,892	-77%	0	0	—
case-21 Retrieval-Augmented Generation evaluation on ChatRAG Bench shows variance across publications due to differing retrieval context lengths, chunking strategies, and answer overlap metrics. Conduct a forensic protocol investigation of 20 ChatRAG Bench papers and output YAML.	fail→pass	45,271	49,791	+10%	1	1	0%	8,239	9,217	+12%	0	0	—
case-22 We need to run the standard HumanEval evaluation harness on our fine-tuned Llama-3 8B model checkpoint using temperature 0.2 and top_p 0.95. Generate a Python script using the official human-eval framework to execute the benchmark and record pass@1 scores.	fail→pass	42,404	22,345	-47%	1	1	0%	2,629	4,523	+72%	0	0	—
case-23 We want to fine-tune a Mistral-7B model on the MATH dataset to improve competition math reasoning performance. Provide a PyTorch or HuggingFace Transformers training script with LoRA configuration and SFTTrainer hyperparameters.	pass→pass	19,857	25,890	+30%	1	1	0%	4,218	6,114	+45%	0	0	—
case-24 Summarize the key architectural innovations and main benchmark results of the Llama 3 technical report, focusing on its pre-training dataset size and attention mechanisms.	pass→pass	16,770	23,988	+43%	1	1	0%	2,531	5,548	+119%	0	0	—

These cases were also screened by the skill’s intent / safety pass.See the Trust review ↗Reproduce this benchmark in Colab ↗

How this skill was measured

DecimalAI ran this skill against gemini-3.6-flash twice over the same eval suite — once with the skill loaded and once without — and compared the two runs case by case. 24 cases were attempted, and 16 counted toward the lift figure. The other 8 produced results that are not comparable between the two arms, so they are excluded from the headline rather than averaged into it. The headline lift of +25 percentage points is the difference between those two pass rates over the 16 comparable cases. 1 case got worse with the skill loaded, and it is included in that figure.

Model measured: gemini-3.6-flash
Graded by: gemini-3.6-flash
Grading method: judged
Verification: verified
Pass rate without the skill: 12.5%
Pass rate with the skill: 37.5%
Cases attempted: 24
Cases counted toward lift: 16
Cases that regressed: 1
Skill version measured: v1
Run completed: 9/4/2026, 04:55 UTC

Measured example — same prompt, with vs. without the skill · from the benchmark run · case case-09

Prompt

HumanEval+ evaluates LLM code generation using extended test suites, but papers differ in prompt formatting, docstring inclusion, and execution timeout settings. Conduct a protocol forensics analysis across 20 papers on HumanEval+ and return the YAML output.

Without the skill loaded, the model failed this case. With it loaded, the same prompt on the same model passed. This is one improved case from the latest verified run; every case, including any that regressed, is in the table above.

Show both answers

SkillScore blends 4 signals1 of 4 complete

BenchmarkPass rate WITH skill on a verified run38

Live evalNeeds ≥10 evaluated traces in 30 days

AI ratingNeeds ≥10 rated traces

AdoptionNeeds ≥10 serves across ≥2 orgs in 30 days

Efficiency · separate axis (at equal correctness)+0% turns+93% tokens

Shown for transparency; not credited unless a verified run shows no correctness regression.

Backed by 1 of 4 signals — each added signal makes the score more robust.

Verified on gemini-3.6-flash · re-tested 17 days ago

Other measured skills in the registry, with their headline benchmark lift.