Verify a BibTeX file for hallucinated or fabricated references by cross-checking every entry against CrossRef, arXiv, and DBLP. Reports each reference as verified, suspect, or not found, with field-level mismatch details (title, authors, year, DOI).
SkillScore82Maintained
Automatic
◆Research pack
Last verified 10/6/2026
What it changes
Passed 16 of 19 test cases with the skill, and 6 of 19 without it.
Without skill32%
With skill82%
Checks it passed that the model alone missed
Identifies litellm as a required package
States that the entry status will be suspect
Identifies py-openjudge as a required package
and 16 more in the full report
12 fixed6 passed either way3 still failing1 broke
Pass-rate lift
+50 pp
Tokens
-32%
Agent turns
0%
Response time
-52%
Model gemini-3.6-flashMeasured 10/6/2026v118 of 22 cases passed with skill · 19 comparable
Tokens, turns and time describe resource use, not a dollar price.
See it in action
One of 10 test cases the skill fixed. Same model, same prompt, with and without the skill.
1 of 10
Prompt
An entry in `arxiv_refs.bib` contains a malformed DOI string that fails to resolve to any published paper during the database match phase. Which specific attribute in the detailed report reflects whether the DOI resolved correctly?
What was checked, without and with the skill
Checked
Without skill
With skill
Identifies doi_match as the field attribute for DOI resolution matching
Missed
Passed
case-19 · gemini-3.6-flash · 10/6/2026 · answers are capped and redacted
What's inside
The instructions your model reads when the skill loads.
SKILL.md66 linessingle file
BibTeX Verification Skill
Check every entry in a .bib file against real academic databases using the
OpenJudge PaperReviewPipeline in BibTeX-only mode:
Parse — extract all entries from the .bib file
Lookup — query CrossRef, arXiv, and DBLP for each reference
Match — compare title, authors, year, and DOI
Report — flag each entry as verified, suspect, or not_found
Prerequisites
pip install py-openjudge litellm
Gather from user before running
Info
Required?
Notes
BibTeX file path
Yes
.bib file to verify
CrossRef email
No
Improves CrossRef API rate limits
Quick start
# Verify a standalone .bib file
python -m cookbooks.paper_review --bib_only references.bib
# With CrossRef email for better rate limits
python -m cookbooks.paper_review --bib_only references.bib --email [email protected]
# Save report to a custom path
python -m cookbooks.paper_review --bib_only references.bib \
--email [email protected] --output bib_report.md
Benchmark LLM reference recommendation capabilities by verifying every cited paper against Crossref, PubMed, arXiv, and DBLP. Measures hallucination rate, per-field accuracy (title/author/year/DOI), discipline breakdown, and year constraint compliance. Supports tool-augmented (ReAct + web search) mode. Use when the user asks to evaluate, benchmark, or compare models on academic reference hallucination, literature recommendation quality, or citation accuracy.
Automatically evaluate and compare multiple AI models or agents without pre-existing test data. Generates test queries from a task description, collects responses from all target endpoints, auto-generates evaluation rubrics, runs pairwise comparisons via a judge model, and produces win-rate rankings with reports and charts. Supports checkpoint resume, incremental endpoint addition, and judge model hot-swap. Use when the user asks to compare, benchmark, or rank multiple models or agents on a cust