Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →yogsoth-ai/evaluation-protocol-comparison
Compare implementation differences of same benchmark across papers
Passed 14 of 25 test cases with the skill, and 6 of 25 without it.
and 12 more in the full report
9 fixed5 passed either way10 still failing1 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 9 test cases the skill fixed. Same model, same prompt, with and without the skill.
Provide a protocol evaluation comparison for SWE-bench across published studies in a YAML format. I need to see how variations in split filtering, prompt framing, max tokens, metric parsing, and container execution affect results, complete with variance levels, impact estimates, and standardization advice.
| Checked | Without skill | With skill |
|---|---|---|
| Assigns variance_level values restricted to none, low, medium, high, or extreme. | Missed | Passed |
| Reports a papers_compared count of at least 8. | Missed | Passed |
The instructions your model reads when the skill loads.
Compare how different papers implement the same benchmark to expose hidden protocol variance that undermines cross-paper score comparability.
Collect 10-15 papers that report results on the target benchmark:
Search queries: "[benchmark name] evaluation", "[benchmark name] results", "[benchmark name] state-of-the-art"
For each paper, run protocol-element-extraction SOP to extract:
| Element Category | Specific Parameters |
|---|---|
| Data | Split version, subset selection, preprocessing, filtering |
| Prompting | Template format, few-shot examples (count, selection), instruction wording |
| Generation | Decoding strategy, temperature, top-p/top-k, max tokens, stop criteria |
| Evaluation | Metric implementation, postprocessing, normalization, scoring script version |
| Infrastructure | Framework, precision (fp16/bf16/fp32), batch size, hardware |
Build a comparison matrix:
Compute per-element variance:
test cases passed, vs 6 / 25 without
Published skills whose descriptions closely match this one, with their headline benchmark lift.
More skills from the same source and nearby in the registry, with their headline benchmark lift.