Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →brycewang-stanford/cikm-reproducibility
Use when hardening the reproducibility of a CIKM paper — pinning the pipeline stages where IR, mining, and knowledge-management results silently diverge, documenting KGs and enterprise data that cannot be released, keeping the GenAI disclosure consistent with how code and data were produced, and preparing the post-acceptance release.
Passed 18 of 22 test cases with the skill, and 12 of 22 without it.
and 7 more in the full report
7 fixed11 passed either way3 still failing1 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 7 test cases the skill fixed. Same model, same prompt, with and without the skill.
For our model training experiments, we ran 50 random seeds and chose the single best performing seed (42) for Table 1. In our paper, we plan to simply report 'models were trained using seed 42' without mentioning the 50-seed selection process. Is this reporting standard accurate?
| Checked | Without skill | With skill |
|---|---|---|
| The response requires documenting the seed policy in the protocol. | Passed | Passed |
| The response requires documenting the model selection rule in the protocol. | Missed | Passed |
The instructions your model reads when the skill loads.
Reproducibility at CIKM has a venue-specific difficulty: the typical paper chains components from different communities — an index, a graph, a model, a ruleset — and each link has its own silent-divergence habits. A reader who cannot rebuild the chain cannot attribute the result, and a blended review panel contains someone able to notice each weak link.
| Chain link | How results silently drift | Pin |
|---|---|---|
| Text preprocessing / indexing | Tokenizer versions, stopword lists, index-time defaults differ across toolkits | Name toolkit + version + config file in the artifact |
| KG snapshot | Public KGs (Wikidata-class) change daily; entity counts drift | Freeze and state the dump date; ship the extracted subgraph if licensable |
| Candidate generation | Recall stage caps and thresholds rarely reported | Report every cutoff; they bound the final metrics |
| Training | Seeds, hardware nondeterminism, early-stopping criteria | Seed policy + selection rule in the protocol paragraph |
| Evaluation | Metric implementations disagree at tie-breaking and cutoffs | Name the evaluation library version; never hand-roll silently |
| LLM components | Model version/API drift; prompts unlogged | Pin model identifiers and dates; log prompts verbatim in the artifact |
The discipline: for each link, either the artifact pins it or the paper states it. A link pinned nowhere is where a failed replication will land.
CIKM's KM lane routinely involves enterprise corpora, clickstreams, or proprietary KGs that cannot ship. The venue-honest pattern:
CIKM 2026's mandatory GenAI Usage Disclosure covers code and data, not just prose (source map, 2026-07-08). Treat it as part of the methods record: if evaluation
test cases passed, vs 12 / 22 without
Published skills whose descriptions closely match this one, with their headline benchmark lift.
More skills from the same source and nearby in the registry, with their headline benchmark lift.