Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

Novelty Check Skill · brycewang-stanford | DecimalAI

brycewang-stanford/novelty-check

SkillScore59Maintained

Verify research idea novelty against recent literature. Use when user says "查新", "novelty check", "有没有人做过", "check novelty", or wants to verify a research idea is novel before implementing.

researchby brycewang-stanford

Links this skill to your agents and keeps it current with the author’s updates.

Automatic

Research pack

1 · One-time — install the CLI (docs ↗)

pip install decimalai

2 · Get the skill — writes .claude/skills/brycewang-stanford-novelty-check/SKILL.md

decimalai skills pull brycewang-stanford-novelty-check --out .claude/skills/

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/brycewang-stanford-novelty-check/SKILL.md https://app.decimal.ai/s/brycewang-stanford-novelty-check/SKILL.md

Raw file: /s/brycewang-stanford-novelty-check/SKILL.md ↗ · machine-readable summary at /s/brycewang-stanford-novelty-check.json ↗

In productionGlobal

how this skill behaves on real traffic · last 30 days

No production data for this skill yet — usage numbers appear here once agents start running it.

Anonymized aggregate across every org using this skill — proof from real production use, not a demo.

Does this skill help? · Benchmark: with vs. withoutsign in to run this on your agents

22 test cases from the publisher’s eval.yaml · each run twice on gemini-3.6-flash — with the skill injected, then without · last run 48d ago

—+7% tokens0% turns59% of whole cases pass with skill (13/22)

Test case	Without → With	Effect	Δ tokens	Δ turns
case-12	✗→✓	▲ Improved	-36%	0%
case-07	✗→✓	▲ Improved	-30%	0%
case-17	✗→✓	▲ Improved	-18%	0%
case-19	✗→✓	▲ Improved	-27%	0%
case-22	✗→✓	▲ Improved	-8%	0%

Showing 5 of 22 cases

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v4 — 8/9/2026

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

brycewang-stanford / Auto-Empirical-Research-Skills

skills/67-econfin-workflow-toolkit/novelty-check/SKILL.md

View on GitHub ↗

License NOASSERTIONImported 8/9/2026Version v4

Skill contents

SKILL.md

Novelty Check Skill

Check whether a proposed method/idea has already been done in the literature: $ARGUMENTS

Constants

REVIEWER_MODEL = gpt-5.4 — Model used via a secondary Codex agent. Must be an OpenAI model (e.g., gpt-5.4, o3, gpt-4o)

Instructions

Given a method description, systematically verify its novelty:

Phase A: Extract Key Claims

Read the user's method description
Identify 3-5 core technical claims that would need to be novel:

What is the method?
What problem does it solve?
What is the mechanism?
What makes it different from obvious baselines?

Phase B: Multi-Source Literature Search

For EACH core claim, search using ALL available sources:

Web Search (via WebSearch):

Search arXiv, Google Scholar, Semantic Scholar
Use specific technical terms from the claim
Try at least 3 different query formulations per claim
Include year filters for 2024-2026

Known paper databases: Check against:

ICLR 2025/2026, NeurIPS 2025, ICML 2025/2026
Recent arXiv preprints (2025-2026)

Read abstracts: For each potentially overlapping paper, WebFetch its abstract and related work section

Phase C: Cross-Model Verification

Call REVIEWER_MODEL via spawn_agent (spawn_agent) with xhigh reasoning:

reasoning_effort: xhigh

Prompt should include:

The proposed method description
All papers found in Phase B
Ask: "Is this method novel? What is the closest prior work? What is the delta?"

Phase D: Novelty Report

Output a structured report:

markdown
## Novelty Check Report

### Proposed Method
[1-2 sentence description]

### Core Claims
1. [Claim 1] — Novelty: HIGH/MEDIUM/LOW — Closest: [paper]
2. [Claim 2] — Novelty: HIGH/MEDIUM/LOW — Closest: [paper]
...

### Closest Prior Work
| Paper | Year | Venue | Overlap | Key Difference |
|-------|------|-------|---------|----------------|

### Overall Novelty Assessment
- Score: X/10
- Recommendation: PROCEED / PROCEED WITH CAUTION / ABANDON
- Key differentiator: [what makes this unique, if anything]
- Risk: [what a reviewer would cite as prior work]

### Suggested Positioning
[How to frame the contribution to maximize novelty perception]

Important Rules

Be BRUTALLY honest — false novelty claims waste months of research time
"Applying X to Y" is NOT novel unless the application reveals surprising insights
Check both the method AND the experimental setting for novelty
If the method is not novel but the FINDING would be, say so explicitly
Always check the most recent 6 months of arXiv — the field moves fast

Illustrative Want to see what brycewang-stanford/novelty-check would put in front of your model? Preview the rendered body and check its trigger phrases — no LLM call, no signup.

Latest benchmark · publisher runVerified

8/11/2026, 04:53 UTC · 22 cases

Measured on 17 comparable cases — below the 22 needed to publish a lift, so this result is shown here but not on the registry card or leaderboard.

Lift

+9%

59% vs 50%

Agent turns

1.0 vs 1.0

Tokens

+7%

2,008.24 vs 1,882.41

fail → pass · 5pass → pass · 8fail → fail · 6pass → fail · 3

Per-case results (22)

Case	Status	Duration (ms)			Turns			Tokens			Tool calls
Case	Status	Without	With	Δ	Without	With	Δ	Without	With	Δ	Without	With	Δ
case-20 When conducting multi-source literature verification for a reinforcement learning contribution, what paper databases and conference venues must be searched?	pass→pass	13,772	7,910	-43%	1	1	0%	2,571	2,411	-6%	0	0	—
case-14 I applied standard stochastic gradient descent to a transformer model and discovered an unexpected phase transition phenomenon in token embeddings. The training method itself is standard, but the empirical phenomenon is new. How should this scenario be evaluated?	pass→fail	13,735	6,124	-55%	1	1	0%	2,493	1,192	-52%	0	0	—
case-12 My custom evaluation script outputs recommendation status options like 'APPROVED', 'NEEDS REVISION', or 'REJECTED'. What exact recommendation labels should be assigned in the overall novelty assessment?	fail→pass	9,234	2,249	-76%	1	1	0%	1,881	1,196	-36%	0	0	—
case-01 I am planning a paper on a dynamic pruning algorithm for vision transformers that adjusts sparsity based on per-token activation variance during inference. Could you run a comprehensive literature search to verify if this concept has been done before? Please break down my main technical claims with individual novelty assessments, compile a comparison table of the closest existing papers from recent conferences/arXiv, provide an overall novelty score and recommendation, and suggest how I should frame the paper's contribution.	fail→fail	22,339	5,766	-74%	1	1	0%	3,856	1,081	-72%	0	0	—
case-02 Can you evaluate the novelty of my research idea? I want to use direct preference optimization (DPO) combined with contrastive decoding to reduce hallucination in medical LLMs. Search recent literature and preprints across major ML venues. I need a report that lists my core claims with their uniqueness levels, displays a table comparing closest prior works, gives a final novelty rating with reviewer risks, and provides advice on how to position my method.	fail→fail	17,397	5,416	-69%	1	1	0%	3,372	1,085	-68%	0	0	—
case-13 I am taking the existing LoRA fine-tuning method and applying it without changes to audio classification on ESC-50. A naive reviewer might rate this high novelty because no paper used LoRA on ESC-50 specifically. How should an idea that simply applies method X to domain Y be evaluated for novelty?	pass→fail	14,741	6,552	-56%	1	1	0%	2,700	1,182	-56%	0	0	—
case-03 I've designed a novel loss function for diffusion models that penalizes high-frequency noise artifacts during early reverse-diffusion steps. Before writing the draft, I need a rigorous novelty audit. Please evaluate the novelty of this idea across recent publications, outline the core claims with specific novelty ratings, summarize the most similar papers in a structured table, give an overall score with proceed/caution recommendations, and detail how to best position this work against existing baselines.	fail→fail	18,479	7,564	-59%	1	1	0%	3,557	1,180	-67%	0	0	—
case-04 I am submitting a paper to NeurIPS 2025 and need my raw text converted into a clean LaTeX document using standard package imports like hyperref and amsmath. Here is my draft text: 'We present a linear-time attention mechanism for sequence modeling.' Can you generate the complete LaTeX manuscript boilerplate?	pass→pass	10,616	9,424	-11%	1	1	0%	2,099	2,762	+32%	0	0	—
case-05 Please proofread my submitted research paper abstract for grammatical clarity, passive voice usage, and typographical errors: 'In this work we introduces a dynamic routing tree that reduce parameter counts by 40 percentage while maintaining accuracy standard on ImageNet.'	pass→fail	6,801	25,518	+275%	1	1	0%	1,401	4,112	+194%	0	0	—
case-06 We are designing a user study to compare two code generation tools (Tool A vs Tool B) on task completion time. Assuming a standard statistical significance level alpha of 0.05 and desired power of 0.80 with an estimated Cohen's d effect size of 0.5, what is the required sample size per group?	pass→pass	7,574	11,836	+56%	1	1	0%	1,591	3,362	+111%	0	0	—
case-07 I am evaluating a novel sparse attention mechanism for vision transformers. When running a cross-model verification step to check novelty against published baselines, I usually just ask a fast default model like gpt-4o-mini. What secondary model and reasoning configuration should be used for this secondary verification step?	fail→pass	14,820	6,335	-57%	1	1	0%	2,682	1,875	-30%	0	0	—
case-08 I have a new graph neural network method for molecular binding prediction. I tend to list 10-15 detailed feature claims when auditing novelty. How many core technical claims should be extracted from the method description?	pass→pass	12,285	4,095	-67%	1	1	0%	2,131	1,334	-37%	0	0	—
case-09 When searching literature databases for prior work on my contrastive reward model claim, I usually type a single broad search string into Google Scholar. How many query formulations should be tried per claim?	pass→pass	10,139	3,410	-66%	1	1	0%	1,805	1,373	-24%	0	0	—
case-10 I am checking whether my multi-modal alignment method is novel. Should I filter paper databases across the entire history from 2010 to 2023, or focus on a specific recent date window for conferences like ICLR/NeurIPS/ICML and preprints?	fail→fail	12,695	8,660	-32%	1	1	0%	2,196	2,244	+2%	0	0	—
case-11 I am preparing a literature comparison table for a soft prompt tuning method. Standard reviews list columns: Author, Title, Summary, Pros, Cons. What exact table columns should be used to summarize closest prior work?	fail→fail	13,001	12,131	-7%	1	1	0%	2,257	2,941	+30%	0	0	—
case-15 When evaluating whether a proposed benchmark evaluation strategy is original, should the audit evaluate only the core algorithm, or does the search scope include the experimental setting as well?	pass→pass	11,654	8,447	-28%	1	1	0%	1,876	1,988	+6%	0	0	—
case-16 I am assessing two core claims for my low-rank adaptation variant: claim 1 is low-rank factorization of key vectors, and claim 2 is dynamic rank allocation based on gradient norm. How should individual claim novelty levels be rated in the core claims section?	pass→pass	13,811	10,916	-21%	1	1	0%	2,474	2,645	+7%	0	0	—
case-17 In the overall novelty assessment section of my research evaluation report, what specific risk factor should be highlighted for peer review?	fail→pass	11,174	4,022	-64%	1	1	0%	1,790	1,461	-18%	0	0	—
case-18 When providing the overall novelty score in the assessment report, what numerical rating scale should be used?	fail→fail	7,792	1,688	-78%	1	1	0%	1,337	946	-29%	0	0	—
case-19 When initiating the secondary reviewer model prompt during cross-model verification, what three specific questions must be asked about the proposed idea?	fail→pass	7,814	1,647	-79%	1	1	0%	1,378	1,003	-27%	0	0	—
case-21 What is the explicit goal of the suggested positioning section at the end of the novelty check report?	pass→pass	6,555	2,798	-57%	1	1	0%	1,099	1,164	+6%	0	0	—
case-22 What top-level markdown section headers must appear in the completed report for a proposed deep learning architecture novelty audit?	fail→pass	8,115	3,328	-59%	1	1	0%	1,433	1,323	-8%	0	0	—

These cases were also screened by the skill’s intent / safety pass.See the Trust review ↗Reproduce this benchmark in Colab ↗

How this skill was measured

DecimalAI ran this skill against gemini-3.6-flash twice over the same eval suite — once with the skill loaded and once without — and compared the two runs case by case. 22 cases were attempted, and 17 counted toward the lift figure. The other 5 produced results that are not comparable between the two arms, so they are excluded from the headline rather than averaged into it. The headline lift of +9 percentage points is the difference between those two pass rates over the 17 comparable cases. 5 cases got worse with the skill loaded, and they are included in that figure.

Model measured: gemini-3.6-flash
Graded by: gemini-3.6-flash
Grading method: judged
Verification: verified
Pass rate without the skill: 50%
Pass rate with the skill: 59.1%
Cases attempted: 22
Cases counted toward lift: 17
Cases that regressed: 5
Skill version measured: v4
Run completed: 8/11/2026, 04:53 UTC

Measured example — same prompt, with vs. without the skill · from the benchmark run · case case-12

Prompt

My custom evaluation script outputs recommendation status options like 'APPROVED', 'NEEDS REVISION', or 'REJECTED'. What exact recommendation labels should be assigned in the overall novelty assessment?

Without the skill loaded, the model failed this case. With it loaded, the same prompt on the same model passed. This is one improved case from the latest verified run; every case, including any that regressed, is in the table above.

Show both answers

SkillScore blends 4 signals1 of 4 complete

BenchmarkPass rate WITH skill on a verified run59

Live evalNeeds ≥10 evaluated traces in 30 days

AI ratingNeeds ≥10 rated traces

AdoptionNeeds ≥10 serves across ≥2 orgs in 30 days

Efficiency · separate axis (at equal correctness)+0% turns+9% tokens

Shown for transparency; not credited unless a verified run shows no correctness regression.

Backed by 1 of 4 signals — each added signal makes the score more robust.

Verified on gemini-3.6-flash · re-tested 47 days ago

Model	Method	Date	Lift
gemini-3.6-flash	verified	8/8/2026	—

Other measured skills in the registry, with their headline benchmark lift.