Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →SOTA Performance Baseline Campaign — 5 strategies for systematically collecting, standardizing, and analyzing performance data across methods. Produces standardized comparison tables, progress curves, and headroom analysis.
.claude/skills/yogsoth-ai-baseline-establishment/SKILL.md| Test case | Without → With | Effect | Δ tokens | Δ turns |
|---|---|---|---|---|
| case-06 | ✗→✓ | ▲ Improved | -3% | 0% |
| case-07 | ✗→✓ | ▲ Improved | -35% | 0% |
| case-08 | ✗→✓ | ▲ Improved | -27% | 0% |
| case-10 | ✗→✓ | ▲ Improved | 18% | 0% |
| case-11 | ✗→✓ | ▲ Improved | 127% | 0% |
| User Intent | Route To | |-------------|----------| | Find all methods for a task | method-inventory | | Extract scores from papers | performance-extraction | | Normalize conditions across papers | condition-standardization | | Check reproducibility / discrepancies | discrepancy-analysis | | Track progress over time / headroom | progress-quantification |
| Strategy | Purpose | |----------|---------| | method-inventory | Comprehensively identify all relevant methods for a task | | performance-extraction | Systematically extract performance data and conditions from papers | | condition-standardization | Standardize evaluation condition differences across papers | | discrepancy-analysis | Identify discrepancies between reported and reproducible scores | | progress-quantification | Track performance progress over time, quantify remaining headroom |
| Tactic | Purpose | |--------|---------| | leaderboard-harvesting | Systematically collect performance data from platforms and papers | | condition-normalization | Compare and standardize experimental conditions across papers | | progress-curve-construction | Build performance-over-time progress curves |
| SOP | Purpose | |-----|---------| | method-discovery | Identify methods via literature, leaderboards, citation chains | | score-extraction | Extract (Task, Dataset, Metric, Score, Conditions) tuples | | condition-cataloging | Record evaluation conditions per method | | reproducibility-checklist-audit | Assess paper against ML Reproducibility Checklist | | performance-table-assembly | Assemble unified comparison table | | compute-normalization | Normalize results by compute budget | | discrepancy-identification | Compare same-method scores across sources | | headroom-estimation | Estimate ceiling vs current SOTA gap | | progress-curve-fitting | Construct performance-over-time data | | baseline-synthesis | Produce final structured baseline report |
| Strategy | Methods | Data Points | Web Searches | |----------|---------|-------------|--------------| | method-inventory | 50 | 0 | 60 | | performance-extraction | 30 | 150 | 40 | | condition-standardization | 20 | 60 | 30 | | discrepancy-analysis | 15 | 45 | 30 | | progress-quantification | 30 | 100 | 40 | | TOTAL | 145 | 355 | 200 |
| MCP Server | Tools | |------------|-------| | brave-search | brave_web_search, brave_llm_context | | apify | rag-web-browser, google-scholar-scraper | | alphaxiv | get_paper_content, answer_pdf_queries | | semantic-scholar | ss_paper, ss_relevance_search, ss_citations, ss_references |
Campaign outputs are accumulated in the calling knowledge-acquisition context:
methods_inventory.json — All discovered methods with metadataperformance_data.json — Extracted scores with provenanceconditions_matrix.json — Standardized conditions per methoddiscrepancy_report.json — Flagged score inconsistenciesprogress_curves.json — Time-series performance databaseline_report.md — Final synthesized baseline document<!-- BEGIN available-tables (generated) -->
Optional, no fixed order; the final leaf is always a sop.
| Strategy | When to use | | --- | --- | | condition-standardization | Standardize evaluation condition differences across papers — 20 methods, 60 data points, 30 web searches budget | | discrepancy-analysis | Identify discrepancies between reported and reproducible scores — 15 methods, 45 data points, 30 web searches budget | | method-inventory | Comprehensively identify all relevant methods for a task — 50 methods, 60 web searches budget | | performance-extraction | Systematically extract performance data and conditions from papers — 30 methods, 150 data points, 40 web searches budget | | progress-quantification | Track performance progress over time, quantify remaining headroom — 30 methods, 100 data points, 40 web searches budget |
Optional, no fixed order; the final leaf is always a sop.
| SOP | When to use | | --- | --- | | context-checkpoint | Append research process and results to the current Phase's context file. Covers both process and results with genuine substance. Use this skill at plan-designated checkpoint points — typically after each strategy completes or at key decision nodes within a research Phase. | | context-init | Create a new context file for a research Phase. Called once at Phase start to initialize the file that subsequent context-checkpoint calls will append to. Use this skill whenever a new research Phase begins and a fresh context file is needed. |
<!-- END available-tables (generated) -->
| Case | Status | Duration (ms) | Turns | Tokens | Tool calls | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Without | With | Δ | Without | With | Δ | Without | With | Δ | Without | With | Δ | ||
case-01 | fail→fail | 33,662 | 5,688 | -83% | 1 | 1 | 0% | 5,843 | 1,571 | -73% | 0 | 0 | — |
case-02 | fail→fail | 29,830 | 6,748 | -77% | 1 | 1 | 0% | 4,975 | 1,594 | -68% | 0 | 0 | — |
case-03 | fail→fail | 6,529 | 7,923 | +21% | 1 | 1 | 0% | 1,016 | 2,258 | +122% | 0 | 0 | — |
case-04 | fail→fail | 18,763 | 26,027 | +39% | 1 | 1 | 0% | 4,018 | 6,718 | +67% | 0 | 0 | — |
case-05 | fail→fail | 16,854 | 19,143 | +14% | 1 | 1 | 0% | 2,786 | 4,355 | +56% | 0 | 0 | — |
case-06 | fail→pass | 20,599 | 9,931 | -52% | 1 | 1 | 0% | 2,829 | 2,758 | -3% | 0 | 0 | — |
case-07 | fail→pass | 15,489 | 3,535 | -77% | 1 | 1 | 0% | 2,665 | 1,743 | -35% | 0 | 0 | — |
case-08 | fail→pass | 16,526 | 4,302 | -74% | 1 | 1 | 0% | 2,553 | 1,867 | -27% | 0 | 0 | — |
case-09 | fail→fail | 16,380 | 2,033 | -88% | 1 | 1 | 0% | 2,770 | 1,463 | -47% | 0 | 0 | — |
case-10 | fail→pass | 28,033 | 1,613 | -94% | 1 | 1 | 0% | 1,199 | 1,410 | +18% | 0 | 0 | — |
case-11 | fail→pass | 4,795 | 3,157 | -34% | 1 | 1 | 0% | 758 | 1,723 | +127% | 0 | 0 | — |
case-12 | pass→pass | 11,253 | 3,233 | -71% | 1 | 1 | 0% | 1,674 | 1,666 | -0% | 0 | 0 | — |
case-13 | pass→pass | 10,368 | 2,380 | -77% | 1 | 1 | 0% | 1,575 | 1,554 | -1% | 0 | 0 | — |
case-14 | pass→pass | 10,030 | 2,626 | -74% | 1 | 1 | 0% | 1,486 | 1,576 | +6% | 0 | 0 | — |
case-20 | fail→pass | 8,720 | 2,336 | -73% | 1 | 1 | 0% | 1,344 | 1,545 | +15% | 0 | 0 | — |
case-15 | fail→pass | 9,337 | 2,757 | -70% | 1 | 1 | 0% | 1,510 | 1,559 | +3% | 0 | 0 | — |
case-16 | fail→pass | 18,309 | 9,416 | -49% | 1 | 1 | 0% | 2,862 | 2,706 | -5% | 0 | 0 | — |
case-17 | fail→fail | 20,587 | 1,764 | -91% | 1 | 1 | 0% | 852 | 1,372 | +61% | 0 | 0 | — |
case-18 | pass→pass | 15,096 | 2,874 | -81% | 1 | 1 | 0% | 2,172 | 1,547 | -29% | 0 | 0 | — |
case-19 | fail→pass | 16,581 | 3,315 | -80% | 1 | 1 | 0% | 2,544 | 1,636 | -36% | 0 | 0 | — |
case-21 | fail→pass | 15,789 | 1,716 | -89% | 1 | 1 | 0% | 2,433 | 1,337 | -45% | 0 | 0 | — |
case-22 | fail→pass | 7,954 | 1,996 | -75% | 1 | 1 | 0% | 1,235 | 1,467 | +19% | 0 | 0 | — |
DecimalAI ran this skill against gemini-3.6-flash twice over the same eval suite — once with the skill loaded and once without — and compared the two runs case by case. 22 cases were attempted, and 18 counted toward the lift figure. The other 4 produced results that are not comparable between the two arms, so they are excluded from the headline rather than averaged into it. The headline lift of +50 percentage points is the difference between those two pass rates over the 18 comparable cases.
Without the skill loaded, the model failed this case. With it loaded, the same prompt on the same model passed. This is one improved case from the latest verified run; every case, including any that regressed, is in the table above.
Other measured skills in the registry, with their headline benchmark lift.