Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

Get Started Free →

Discrepancy Analysis · yogsoth-ai | DecimalAI

yogsoth-ai/discrepancy-analysis

NewSkillScoreMaintained

Identify discrepancies between reported and reproducible scores — 15 methods, 45 data points, 30 web searches budget

researchby yogsoth-ai

Links this skill to your agents and keeps it current with the author’s updates.

Automatic

Research pack

1 · One-time — install the CLI (docs ↗)

pip install decimalai

2 · Get the skill — writes .claude/skills/yogsoth-ai-discrepancy-analysis/SKILL.md

decimalai skills pull yogsoth-ai-discrepancy-analysis --out .claude/skills/

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/yogsoth-ai-discrepancy-analysis/SKILL.md https://app.decimal.ai/s/yogsoth-ai-discrepancy-analysis/SKILL.md

Raw file: /s/yogsoth-ai-discrepancy-analysis/SKILL.md ↗ · machine-readable summary at /s/yogsoth-ai-discrepancy-analysis.json ↗

In productionGlobal

how this skill behaves on real traffic · last 30 days

No production data for this skill yet — usage numbers appear here once agents start running it.

Anonymized aggregate across every org using this skill — proof from real production use, not a demo.

Does this skill help? · Benchmark: with vs. withoutsign in to run this on your agents

22 test cases from the publisher’s eval.yaml · each run twice on gemini-3.6-flash — with the skill injected, then without · last run 6d ago

—+141% tokens0% turns32% of whole cases pass with skill (7/22)

Test case	Without → With	Effect	Δ tokens	Δ turns
case-22	✗→✓	▲ Improved	289%	0%
case-16	✗→✓	▲ Improved	280%	0%
case-18	✗→✓	▲ Improved	153%	0%
case-20	✗→✓	▲ Improved	83%	0%
case-08	✓→✗	▼ Worse	282%	0%

Showing 5 of 22 cases

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v4 — 8/3/2026

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

yogsoth-ai / de-anthropocentric-research-engine

skills/discrepancy-analysis/SKILL.md

View on GitHub ↗

License Apache-2.0Imported 8/3/2026Version v1

Skill contents

SKILL.md

Discrepancy Analysis

Purpose

Detect inconsistencies between scores reported in original papers versus third-party reproductions, leaderboard entries, and ablation studies. Flags methods with suspicious performance claims, identifies common sources of score inflation, and assesses the reliability of reported baselines.

Budget

| Resource | Floor | Target | |----------|-------|--------| | Methods analyzed | 10 | 15 | | Data points compared | 30 | 45 | | Web searches | 20 | 30 | | Reproduction studies consulted | 5 | 10 |

State Ledger

<HARD-GATE>
| Metric | Current | Target | Status |
|--------|---------|--------|--------|
| Methods analyzed | 0 | 15 | BLOCKED |
| Score pairs compared | 0 | 45 | BLOCKED |
| Discrepancies flagged | 0 | — | — |
| Reproduction studies found | 0 | 10 | — |
| Reliability ratings assigned | 0 | 15 | — |
</HARD-GATE>

Cannot exit until score_pairs_compared >= 36 (80% of target).

Available Tactics

leaderboard-harvesting — Collect multiple sources for cross-validation

Available SOPs

discrepancy-identification — Compare same-method scores across sources
reproducibility-checklist-audit — Assess paper reproducibility completeness

Execution Guidance

For each method, collect scores from multiple independent sources
Use discrepancy-identification SOP to flag significant deviations
Search for reproduction studies, blog posts, and issue trackers
Apply reproducibility-checklist-audit to papers with large discrepancies
Categorize discrepancy sources (data leakage, cherry-picked seeds, unfair baselines)
Assign reliability ratings to each method's reported scores
Document which baselines in the field are trustworthy vs. inflated

Output Format

json
{
  "discrepancies": [
    {
      "method": "string",
      "dataset": "string",
      "metric": "string",
      "reported_score": 0.0,
      "reproduced_score": 0.0,
      "delta": 0.0,
      "delta_significant": true,
      "likely_cause": "string",
      "sources": ["string"]
    }
  ],
  "reliability_ratings": [
    {
      "method": "string",
      "rating": "high|medium|low|unreliable",
      "reproducibility_checklist_score": 0,
      "notes": "string"
    }
  ],
  "systematic_issues": [
    {
      "issue": "string",
      "affected_methods": ["string"],
      "prevalence": "string"
    }
  ]
}

Available Tactics

Optional, no fixed order; the final leaf is always a sop.

| Tactic | When to use | | --- | --- | | leaderboard-harvesting | Systematically collect performance data from platforms and papers |

Available SOPs

Optional, no fixed order; the final leaf is always a sop.

| SOP | When to use | | --- | --- | | discrepancy-identification | Compare same-method scores across sources, flag significant deviations | | reproducibility-checklist-audit | Assess paper completeness against ML Reproducibility Checklist |

Want to see if yogsoth-ai/discrepancy-analysis would activate on your phrasing? Try it before you install — no LLM call, no signup.

Latest benchmark · publisher runVerified

8/23/2026, 01:40 UTC · 22 cases

Measured on 13 comparable cases — below the 22 needed to publish a lift, so this result is shown here but not on the registry card or leaderboard.

Lift

-18%

32% vs 50%

Agent turns

0%

1.0 vs 1.0

Tokens

+141%

6,101.77 vs 2,536.23

fail → pass · 4pass → pass · 3fail → fail · 7pass → fail · 8

Per-case results (22)

Case	Status	Duration (ms)			Turns			Tokens			Tool calls
Case	Status	Without	With	Δ	Without	With	Δ	Without	With	Δ	Without	With	Δ
case-07 Analyze score variance for ObjectDetectionNet on COCO val2017 between original paper claims (54.2 mAP) and independent benchmark runs (48.1 mAP). Provide output as a JSON object with keys: discrepancies, reliability_ratings, and systematic_issues.	fail→fail	13,246	28,179	+113%	1	1	0%	2,391	6,995	+193%	0	0	—
case-08 Evaluate claims for GraphConvNet on PubMed graph classification where original paper claims 89.5% accuracy but third-party evaluation gets 81.2%. Output JSON matching schema with discrepancies (method, dataset, metric, reported_score, reproduced_score, delta, delta_significant, likely_cause, sources), reliability_ratings (method, rating, reproducibility_checklist_score, notes), and systematic_issues (issue, affected_methods, prevalence).	pass→fail	9,189	35,607	+287%	1	1	0%	1,840	7,031	+282%	0	0	—
case-21 Evaluate RL-PPO-Agent on MuJoCo HalfCheetah where reported reward is 9800 but 5 reproduction attempts range between 8200 and 9100 due to high seed sensitivity. Return standard discrepancy audit JSON.	pass→pass	12,471	27,367	+119%	1	1	0%	2,565	6,499	+153%	0	0	—
case-22 Analyze BioBERT-v2 score collapse on MedQA (84.0 reported vs 62.0 reproduced) after community found test set questions in the pre-training corpus. Format output as structured audit JSON.	fail→pass	10,339	28,353	+174%	1	1	0%	1,795	6,981	+289%	0	0	—
case-01 I need an audit of recent claims in tabular reinforcement learning papers comparing original published metrics against independent evaluations and open leaderboards. Please analyze these methods, identify score gaps between original reports and third-party attempts, calculate the variance and likely causes, rate each method's reliability along with audit scores, and summarize systematic evaluation issues affecting multiple techniques. Output your findings as a structured JSON object containing discrepancies, reliability ratings, and systematic issues.	fail→fail	25,316	34,804	+37%	1	1	0%	4,656	7,022	+51%	0	0	—
case-02 We are preparing a survey paper on recent vision transformer baseline claims vs third-party benchmark runs. Compare the primary metrics reported by original authors against reproduction repositories and evaluation platforms. Provide a JSON response structured with three top-level keys: a list of flagged metric discrepancies (showing method, dataset, reported/reproduced scores, delta, significance, root cause, sources), a list of reliability ratings for each method with checklist scores, and a list of field-wide systematic issues.	fail→fail	28,209	11,378	-60%	1	1	0%	5,299	1,770	-67%	0	0	—
case-03 Can you evaluate the reported benchmark numbers for popular text embedding models versus community re-evaluations and leaderboard submissions? I need a detailed JSON breakdown that includes specific discrepancy records comparing paper numbers to reproduced values with likely reasons, confidence or reliability ratings for each algorithm alongside reproducibility audit scores, and any common patterns of score inflation or evaluation flaws found across the field.	fail→fail	25,964	12,164	-53%	1	1	0%	5,047	1,787	-65%	0	0	—
case-04 Write a PyTorch Python script to train ResNet-50 on ImageNet and log evaluation accuracy to TensorBoard.	pass→fail	17,663	22,532	+28%	1	1	0%	3,748	5,857	+56%	0	0	—
case-05 Fix a data loading bug in my PyTorch DataLoader script where random seed initialization causes test set data leakage into the training split.	pass→pass	17,550	20,156	+15%	1	1	0%	3,176	4,165	+31%	0	0	—
case-06 Draft the methodology section of a research paper describing a novel sparse attention mechanism for long-context language models.	pass→pass	21,848	18,997	-13%	1	1	0%	3,923	4,402	+12%	0	0	—
case-09 Compare reported vs reproduced metrics for FastSpeech2 on LJSpeech (MOS 4.35 reported vs 4.31 reproduced). Format JSON output using keys discrepancies (method, dataset, metric, reported_score, reproduced_score, delta, delta_significant, likely_cause, sources), reliability_ratings, and systematic_issues.	pass→fail	11,098	8,940	-19%	1	1	0%	2,069	1,288	-38%	0	0	—
case-10 Assess reliability for ModelX whose score dropped 15% due to cherry-picked random seeds. Return JSON with discrepancies, reliability_ratings (method, rating, reproducibility_checklist_score, notes), and systematic_issues. For rating, use one of: high, medium, low, unreliable.	pass→fail	5,473	28,058	+413%	1	1	0%	1,114	6,997	+528%	0	0	—
case-11 Perform audit on LanguageModel-7B benchmark claims across OpenLLM Leaderboard. Return JSON with discrepancies, reliability_ratings (method, rating, reproducibility_checklist_score, notes), and systematic_issues.	fail→fail	11,371	8,125	-29%	1	1	0%	2,259	1,241	-45%	0	0	—
case-12 Audit 5 speech recognition models where training sets inadvertently included test audio clips. Format JSON output with discrepancies, reliability_ratings, and systematic_issues containing issue, affected_methods, and prevalence.	pass→fail	13,643	9,208	-33%	1	1	0%	2,972	1,526	-49%	0	0	—
case-13 Analyze evaluation flaws in recent recommendation system baselines where tuned hyper-parameters were compared against untuned defaults. Format as JSON with discrepancies, reliability_ratings, and systematic_issues (issue, affected_methods, prevalence).	pass→fail	14,767	9,116	-38%	1	1	0%	2,465	1,676	-32%	0	0	—
case-14 Audit tabular classifier TabNet across OpenML benchmarks versus paper claims. Format output as JSON with discrepancies (method, dataset, metric, reported_score, reproduced_score, delta, delta_significant, likely_cause, sources), reliability_ratings, and systematic_issues.	pass→fail	24,832	9,120	-63%	1	1	0%	4,485	1,326	-70%	0	0	—
case-15 Examine why UltraBERT reported 92.1 F1 on SQuAD v2 in paper, but standard evaluation code yields 86.4 F1 when run on non-custom hardware floats. Output JSON with discrepancies, reliability_ratings, and systematic_issues.	fail→fail	12,626	7,228	-43%	1	1	0%	2,114	1,262	-40%	0	0	—
case-16 Evaluate PromptLLM-13B which claimed 78.0 on GSM8K using 5-shot chain-of-thought, whereas third-party evaluation got 61.2 using standard zero-shot prompts. Output JSON with discrepancies, reliability_ratings, and systematic_issues.	fail→pass	10,755	29,390	+173%	1	1	0%	1,840	6,998	+280%	0	0	—
case-17 Audit LLaMA-2-7B MMLU claims (45.3 reported vs 45.1 reproduced across 3 independent repos). Output JSON with discrepancies, reliability_ratings (method, rating, reproducibility_checklist_score, notes), and systematic_issues.	fail→fail	12,251	7,635	-38%	1	1	0%	2,181	1,501	-31%	0	0	—
case-18 Evaluate DiffusionArt model claims on ImageNet generation (FID 2.1 reported vs 3.8 reproduced, IS 220 reported vs 195 reproduced). Format output as JSON with discrepancies, reliability_ratings, and systematic_issues.	fail→pass	8,447	22,131	+162%	1	1	0%	1,577	3,996	+153%	0	0	—
case-19 Review paper claim for SecretNet claiming 99.1% on MNIST without releasing training code or random seeds, where community re-run achieves 92.0%. Output standard audit JSON with discrepancies, reliability_ratings, and systematic_issues.	pass→fail	7,593	28,786	+279%	1	1	0%	1,399	6,987	+399%	0	0	—
case-20 Audit a cluster of 4 NLP papers that reported BLEU-4 using non-standard tokenizers, inflating scores by 4 points compared to sacreBLEU. Return JSON with discrepancies, reliability_ratings, and systematic_issues.	fail→pass	15,959	22,013	+38%	1	1	0%	2,947	5,393	+83%	0	0	—

These cases were also screened by the skill’s intent / safety pass.See the Trust review ↗Reproduce this benchmark in Colab ↗

How this skill was measured

DecimalAI ran this skill against gemini-3.6-flash twice over the same eval suite — once with the skill loaded and once without — and compared the two runs case by case. 22 cases were attempted, and 13 counted toward the lift figure. The other 9 produced results that are not comparable between the two arms, so they are excluded from the headline rather than averaged into it. The headline lift of -18 percentage points is the difference between those two pass rates over the 13 comparable cases. 9 cases got worse with the skill loaded, and they are included in that figure.

Model measured: gemini-3.6-flash
Graded by: gemini-3.6-flash
Grading method: judged
Verification: verified
Pass rate without the skill: 50%
Pass rate with the skill: 31.8%
Cases attempted: 22
Cases counted toward lift: 13
Cases that regressed: 9
Skill version measured: v1
Run completed: 8/23/2026, 01:40 UTC

Measured example — same prompt, with vs. without the skill · from the benchmark run · case case-22

Prompt

Analyze BioBERT-v2 score collapse on MedQA (84.0 reported vs 62.0 reproduced) after community found test set questions in the pre-training corpus. Format output as structured audit JSON.

Without the skill loaded, the model failed this case. With it loaded, the same prompt on the same model passed. This is one improved case from the latest verified run; every case, including any that regressed, is in the table above.

Show both answers

SkillScore blends 4 signalsno signals yet

BenchmarkRun a verified benchmark with positive lift

Live evalNeeds ≥20 evaluated traces in 30 days

AI ratingNeeds ≥10 rated traces

AdoptionNeeds ≥10 serves across ≥2 orgs in 30 days

Efficiency · separate axis (at equal correctness)+0% turns+56% tokens

Shown for transparency; not credited unless a verified run shows no correctness regression.

No signals yet — run a benchmark or collect evaluated traces to earn a score.

Verified on gemini-3.6-flash · re-tested 6 days ago

Other measured skills in the registry, with their headline benchmark lift.