Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

Forecast Accuracy Review · davila7 | DecimalAI

davila7/forecast-accuracy-review

SkillScore82Maintained

Evaluate demand-forecast quality honestly - WMAPE, bias and Forecast Value Added against a naive benchmark over a rolling-origin backtest. Use when the user mentions forecast accuracy, MAPE, demand planning performance, tahmin doğruluğu, talep tahmini, or asks whether a forecasting process or tool is worth it. Differentiator - judges the process (value added vs doing nothing), not just the model.

scienceby davila7

Links this skill to your agents and keeps it current with the author’s updates.

Automatic

Data Analyst pack

1 · One-time — install the CLI (docs ↗)

pip install decimalai

2 · Get the skill — writes .claude/skills/davila7-forecast-accuracy-review/SKILL.md

decimalai skills pull davila7-forecast-accuracy-review --out .claude/skills/

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/davila7-forecast-accuracy-review/SKILL.md https://app.decimal.ai/s/davila7-forecast-accuracy-review/SKILL.md

Raw file: /s/davila7-forecast-accuracy-review/SKILL.md ↗ · machine-readable summary at /s/davila7-forecast-accuracy-review.json ↗

In productionGlobal

how this skill behaves on real traffic · last 30 days

No production data for this skill yet — usage numbers appear here once agents start running it.

Anonymized aggregate across every org using this skill — proof from real production use, not a demo.

Does this skill help? · Benchmark: with vs. withoutsign in to run this on your agents

22 test cases from the publisher’s eval.yaml · each run twice on gemini-3.6-flash — with the skill injected, then without · last run 47d ago

Yes — +32% lift+12% tokens0% turns82% of whole cases pass with skill (18/22)

Test case	Without → With	Effect	Δ tokens	Δ turns
case-04	✗→✓	▲ Improved	25%	0%
case-03	✗→✓	▲ Improved	33%	0%
case-10	✗→✓	▲ Improved	6%	0%
case-12	✗→✓	▲ Improved	-22%	0%
case-13	✗→✓	▲ Improved	2%	0%

Showing 5 of 22 cases

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v4 — 8/3/2026

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

davila7 / claude-code-templates

cli-tool/components/skills/operations/forecast-accuracy-review/SKILL.md

View on GitHub ↗

License MITImported 8/3/2026Version v1

Skill contents

SKILL.md

Forecast Accuracy Review

A forecast is only worth what it adds over the free alternative: shipping last period's number. Every review must answer "how many points does this process add over naive?" before any model discussion.

Required data

Per-SKU demand history at the planning bucket (usually monthly): sku, period, qty. If evaluating an existing forecast, also the forecast values with their creation dates (to avoid hindsight leakage). 18+ periods per SKU for a meaningful backtest; flag SKUs with less.

Workflow

Profile the demand first. Per SKU compute mean, CV and zero-period share; classify smooth / erratic / intermittent / lumpy (defaults: CV 0.5 and 1.0 boundaries, intermittency at >25% zero periods - state them, adjust to natural breaks). Accuracy expectations differ by class; never report one blended number alone.
Set the benchmarks. Naive (last period) always; seasonal naive when 2+ full seasons exist. These are non-negotiable controls.
Backtest rolling-origin. One-step-ahead forecasts for each of the last 6+ periods, expanding window, using only data before each origin. A single train/test split is one lucky draw - do not accept it.
Score with honest metrics:

WMAPE = sum(|error|) / sum(actual) - the volume-weighted headline
Bias = sum(error) / sum(actual) - direction; a fine WMAPE with persistent bias is quietly building excess stock or stockouts
MAPE only as a footnote, and always disclose how many zero-actual periods it dropped

Deliver the FVA verdict. FVA = WMAPE(naive) - WMAPE(candidate), per segment and overall. Negative FVA means the process destroys value - say it plainly.
Validate. Recompute WMAPE for one model directly from the raw backtest rows and confirm it matches the table before presenting.

Pitfalls to check explicitly

MAPE with zeros: undefined on zero-actual periods; silently dropping them fakes precision on intermittent SKUs.
MAPE asymmetry rewards under-forecasting (errors capped at 100% below, unbounded above).
Aggregation mix: a good total can hide terrible A-item accuracy; always show the value-weighted cut.
Lumpy segments: if WMAPE > ~100%, the honest recommendation is an inventory-policy answer (buffers, MTO), not a better model.
Hindsight leakage: forecasts must predate actuals; check timestamps when auditing an existing process.

Output format

Scoreboard table: model x (WMAPE, bias, MAPE-footnote), sorted by WMAPE
FVA statement: "the process adds/destroys X points vs naive" - overall and per segment
Segment table (pattern x best approach)
Two or three recommendation sentences tied to segments, not globals

Worked example with five baseline models and charts: https://github.com/gulmezeren2-byte/forecast-accuracy-lab

Source: industrial-engineering-ai-skills by Eren Gulmez (MIT). The full method pack - entry skill, role agents, data-hygiene rules and artifact templates - lives there.

Illustrative Want to see what davila7/forecast-accuracy-review would put in front of your model? Preview the rendered body and check its trigger phrases — no LLM call, no signup.

Latest benchmark · publisher runVerified

8/12/2026, 21:13 UTC · 22 cases

Lift

+32%

82% vs 50%

Agent turns

1.0 vs 1.0

Tokens

+12%

2,849.18 vs 2,537.09

fail → pass · 9pass → pass · 9fail → fail · 2pass → fail · 2

Per-case results (22)

Case	Status	Duration (ms)			Turns			Tokens			Tool calls
Case	Status	Without	With	Δ	Without	With	Δ	Without	With	Δ	Without	With	Δ
case-06 Our data team performed a backtest on 18 months of history by doing a single 80/20 train/test split to benchmark our demand models. Evaluate this validation methodology and explain how to structure the evaluation windows properly.	pass→pass	15,062	14,216	-6%	1	1	0%	2,880	3,457	+20%	0	0	—
case-04 We are profiling demand for 100 SKUs at Apex Logistics. Modelers suggested marking any SKU with at least 10% zero-demand months as intermittent. Evaluate this classification rule and provide the standard benchmark boundaries for categorizing smooth versus erratic and intermittent demand.	fail→pass	12,507	12,893	+3%	1	1	0%	2,578	3,214	+25%	0	0	—
case-05 Our statistical ensemble model achieved a WMAPE of 34% across regional warehouses, whereas simply using the previous month's actual sales yielded a WMAPE of 29%. Write the Forecast Value Added verdict for management.	pass→pass	9,203	7,466	-19%	1	1	0%	1,633	2,147	+31%	0	0	—
case-01 I have 24 months of monthly demand and forecast history for 30 SKUs across our distribution centers. Could you evaluate our current forecasting process against simple baseline models? I'd like a complete audit including a primary model scoreboard table sorted by overall volume error showing bias and zero-actual notes, an explicit statement on whether our process adds or loses points against the naive benchmark overall and by segment, a summary table mapping demand patterns to the best approach, and 2 to 3 targeted recommendations tied to specific segments.	fail→fail	27,424	19,870	-28%	1	1	0%	4,807	4,440	-8%	0	0	—
case-02 Here is our SKU-level sales history and model forecasts for the past two years. We need a forecast accuracy review to verify if our statistical models beat a simple last-period baseline. Please produce an evaluation report formatted with four sections: 1) A performance scoreboard table listing each model sorted by volume-weighted accuracy with directional bias and zero-period drop notes; 2) A statement declaring the exact value added or destroyed relative to naive overall and per segment; 3) A segment breakdown table pairing each demand behavior pattern to its ideal method; and 4) Two or three concise recommendations directed at individual segments.	fail→fail	25,144	14,101	-44%	1	1	0%	4,675	3,535	-24%	0	0	—
case-03 Can you audit our demand forecasting performance using our historical SKU dataset of actuals and model predictions? Please perform a backtest comparison and return: 1) A ranked scoreboard table of models sorted by volume error along with bias and zero-demand caveats; 2) A clear verdict statement on how many points our process adds or destroys versus naive prediction across the portfolio and per segment; 3) A segment table categorizing demand patterns against their best forecasting strategy; and 4) A few concrete recommendation sentences tied directly to those segments rather than global averages.	fail→pass	20,227	23,474	+16%	1	1	0%	3,938	5,248	+33%	0	0	—
case-07 During our SKU review, the spare parts segment exhibited lumpy demand patterns with a WMAPE exceeding 110%. The analytics team proposed tuning a complex neural network to reduce error on these SKUs. Evaluate this proposal and recommend the appropriate operational strategy.	pass→pass	17,784	10,702	-40%	1	1	0%	2,938	2,469	-16%	0	0	—
case-08 A planner argues that MAPE should be our primary metric because it is easily understood by executive leadership, even though our inventory includes low-volume items. Explain the exact mathematical distortion that occurs when relying on MAPE for inventory management.	pass→pass	12,508	15,423	+23%	1	1	0%	2,507	3,624	+45%	0	0	—
case-09 Model A has a WMAPE of 18% and a Bias of +12%. Model B has a WMAPE of 20% and a Bias of 0%. Explain why a low WMAPE alone can be dangerous for working capital and stock availability.	pass→pass	10,930	10,734	-2%	1	1	0%	2,083	2,542	+22%	0	0	—
case-10 We have 8 months of sales history for a newly launched product line and want to run a full backtest to measure model accuracy and FVA. How should this dataset be handled in an accuracy review?	fail→pass	15,233	12,586	-17%	1	1	0%	2,700	2,868	+6%	0	0	—
case-11 Provide the exact standard formula for calculating WMAPE across a portfolio of SKUs to ensure our data pipeline computes volume-weighted accuracy correctly.	pass→pass	11,160	8,427	-24%	1	1	0%	2,114	2,483	+17%	0	0	—
case-12 When auditing seasonal consumer goods SKUs, under what specific historical data availability condition must Seasonal Naive be included as a baseline control alongside last-period Naive?	fail→pass	8,148	2,400	-71%	1	1	0%	1,516	1,186	-22%	0	0	—
case-13 When auditing an existing enterprise forecasting process, how do you verify that backtested forecasts did not suffer from hindsight leakage?	fail→pass	16,550	12,785	-23%	1	1	0%	3,008	3,057	+2%	0	0	—
case-14 We completed our model backtest scoring across 500 SKUs and generated summary accuracy tables. What quality assurance validation must be performed directly against raw backtest rows before presenting results to stakeholders?	fail→pass	13,745	8,430	-39%	1	1	0%	2,552	2,226	-13%	0	0	—
case-15 Our corporate supply chain dashboard shows an overall company forecast WMAPE of 12%, but high-value A-items are frequently stocking out. Explain what causes this discrepancy.	pass→pass	12,599	11,433	-9%	1	1	0%	2,247	2,936	+31%	0	0	—
case-16 We want to set up an ABC inventory classification for our warehouse SKUs to prioritize cycle counting frequency. A team member suggested using Coefficient of Variation and zero-demand frequency to segment items into A, B, and C classes. Evaluate this approach and explain how ABC inventory classification should be calculated.	pass→pass	19,731	15,040	-24%	1	1	0%	2,976	3,409	+15%	0	0	—
case-17 Our S&OP committee is establishing our monthly planning process. The demand manager suggests adopting the raw statistical forecast output directly as the final demand plan for operations without gathering inputs from commercial sales or marketing teams. How should the consensus demand planning step in S&OP be executed?	pass→fail	13,012	13,561	+4%	1	1	0%	2,252	3,300	+47%	0	0	—
case-18 Our regional distribution center manages perishable fresh food items with a 14-day expiration shelf life. The inventory team proposes doubling replenishment order quantities to maximize quantity discounts and reduce inbound freight costs. Evaluate this lot-sizing strategy.	fail→pass	13,366	16,137	+21%	1	1	0%	2,461	3,619	+47%	0	0	—
case-19 We are formatting the final model review scoreboard table featuring machine learning models, statistical models, and naive baselines. Which metric determines the primary sort order of the table?	fail→pass	3,650	2,184	-40%	1	1	0%	675	1,062	+57%	0	0	—
case-20 State the exact formula used to compute Forecast Value Added (FVA) for a candidate forecasting process compared against a baseline.	fail→pass	5,546	3,324	-40%	1	1	0%	1,268	1,320	+4%	0	0	—
case-21 An executive asks for a single portfolio-wide overall accuracy percentage number to publish in the annual operations report. How should demand analysts respond regarding single blended accuracy numbers?	pass→fail	11,879	12,675	+7%	1	1	0%	1,967	2,741	+39%	0	0	—
case-22 Provide the exact mathematical definition of forecast Bias used in auditing supply chain model bias.	pass→pass	10,723	5,240	-51%	1	1	0%	2,041	1,799	-12%	0	0	—

These cases were also screened by the skill’s intent / safety pass.See the Trust review ↗Reproduce this benchmark in Colab ↗

How this skill was measured

DecimalAI ran this skill against gemini-3.6-flash twice over the same eval suite — once with the skill loaded and once without — and compared the two runs case by case. 22 cases were attempted. The headline lift of +32 percentage points is the difference between those two pass rates over the 22 comparable cases. 2 cases got worse with the skill loaded, and they are included in that figure.

Model measured: gemini-3.6-flash
Graded by: gemini-3.6-flash
Grading method: judged
Verification: verified
Pass rate without the skill: 50%
Pass rate with the skill: 81.8%
Cases attempted: 22
Cases counted toward lift: 22
Cases that regressed: 2
Skill version measured: v1
Run completed: 8/12/2026, 21:13 UTC

Measured example — same prompt, with vs. without the skill · from the benchmark run · case case-04

Prompt

We are profiling demand for 100 SKUs at Apex Logistics. Modelers suggested marking any SKU with at least 10% zero-demand months as intermittent. Evaluate this classification rule and provide the standard benchmark boundaries for categorizing smooth versus erratic and intermittent demand.

Without the skill loaded, the model failed this case. With it loaded, the same prompt on the same model passed. This is one improved case from the latest verified run; every case, including any that regressed, is in the table above.

Show both answers

SkillScore blends 4 signals1 of 4 complete

BenchmarkPass rate WITH skill on a verified run82

Live evalNeeds ≥10 evaluated traces in 30 days

AI ratingNeeds ≥10 rated traces

AdoptionNeeds ≥10 serves across ≥2 orgs in 30 days

Efficiency · separate axis (at equal correctness)+0% turns+16% tokens

Shown for transparency; not credited unless a verified run shows no correctness regression.

Backed by 1 of 4 signals — each added signal makes the score more robust.

Verified on gemini-3.6-flash · re-tested 46 days ago

Other measured skills in the registry, with their headline benchmark lift.