Forecasting demand when your history is messy

The baseline is not a formality, it is the experiment

The situation is familiar enough to be boring. There is a forecast, and a spreadsheet next to it where a planner adjusts the number before anyone acts on it. Nobody can say how much better the forecast is than the planner alone, because that comparison was never run. Meanwhile a vendor is offering a model that fixes this, and the demo looks convincing.

Before any of that, run the boring comparison: take the value from the same weekday last week and call it the forecast. That is the seasonal naive method; it costs nothing, and scoring it on your holdout windows gives you the number every model has to beat — and the only honest answer to the question your director will eventually ask: what did we get for this?

It is worth knowing how hard that bar is. The M5 Accuracy competition put 7,092 participants on 5,507 teams against real daily retail demand. Per the published results (Makridakis, Spiliotis and Assimakopoulos, International Journal of Forecasting), 2,666 teams (48.4%) beat a plain naive forecast, 1,972 (35.8%) beat seasonal naive, and 415 (7.5%) beat the organizers' strongest benchmark, an exponential smoothing model reconciled bottom-up. Half of a large, capable field did not beat "same as last time."

The finding gets sharper. The paper notes that if teams had submitted their best model rather than the one they chose, 12.2% would have beaten the top benchmark instead of 7.5%. The authors attribute the gap to "misleading validation scores." These teams had built the better model; their evaluation setup could not tell them which one it was. That is a validation failure wearing a modelling failure's clothes — the quietest way a forecasting project goes wrong.

Missing periods: three different things that look the same

Open a demand table with gaps and you are looking at three realities rendered identically:

• True zero. The item was available, nobody ordered it. Real information, and the model should learn it.
• Unobserved. Demand may well have occurred, but the system was down or the feed dropped a partition. The value is unknown, not zero.
• Not yet in existence. The item was not stocked, the facility had not opened, the program had not launched. There is no demand because there was no thing to demand.

Feed all three in as zeros and you have taught the model that this series periodically collapses to nothing and recovers, so it will forecast collapses. Worse, if the unobserved stretches cluster — outages hit month-end, reporting gaps hit holidays — the model learns a seasonal pattern that is a property of your logging, not your demand.

The fix belongs in the data, not the model. Carry an observed flag next to every value. Mask unobserved periods out of the training loss instead of filling them with a number. Truncate the series where the item actually began rather than back-filling zeros to the top of the table. Where a fill is unavoidable, flag it so the backtest can exclude those windows; otherwise you are grading the model on reproducing your imputation.

One trap deserves naming: interpolation is bidirectional. A gap filled with an average of the values either side has used data from after the gap. Do that before splitting, and future information has leaked backwards into every training window that touches it. Backtest accuracy improves, production accuracy does not, and the gap is hard to diagnose months later. Hyndman and Athanasopoulos treat missing values and outliers as a modelling decision rather than a cleanup step.

Regime changes and the assumption nobody writes down

Every forecast rests on an assumption almost never written into the ticket: the process that produced the history is the process that will produce the future. A regime change is what happens when that stops being true. A contract moved volume to another supplier. A facility opened and absorbed traffic from two others. A policy changed who is eligible.

This is a real trade-off with no clean answer. Use more history and more of it comes from a world that no longer exists. Use less and the regime is cleaner, but you may not have enough full cycles left to estimate seasonality. Keep eighteen months of a monthly series and you have one and a half annual cycles — not enough to separate "December is big" from "last December was big."

Automatic changepoint detection helps but does not substitute for knowing what happened. Prophet illustrates the point because its behaviour is documented plainly: it places 25 potential changepoints uniformly in the first 80% of the series and applies a sparse L1-style prior so most stay at zero, with changepoint_prior_scale defaulting to 0.05 and the docs advising you to adjust it if the trend looks over- or under-fit. The 80% default keeps the model from chasing wobbles at the end of the series. Read that carefully: the most recent fifth of your history is excluded from trend-change inference by default — often the exact window containing the change you care about.

So keep a human artifact: a dated event log. What changed, when it took effect, which series it touched, and whether it was one-off or permanent. Ten lines maintained by someone who was in the room outperforms a detector on the same data, because the detector sees only the effect and you can see the cause.

Promotions: the variation you caused on purpose

Demand data differs from most time series in one respect: much of the variation was caused by decisions your own organisation made. A promotion, a price change, a notice sent to beneficiaries. If those drivers are not in the model, it attributes their effects to something it can see, and what it can see is the calendar. It concludes that March is strong. The truth was that you ran a campaign in March. Next March it forecasts a spike you have no plans to produce.

M5 was designed partly to test this. The competition deliberately included exogenous variables such as product prices, promotions and special events alongside the series, and the organizers list "the importance of exogenous/explanatory variables" among the competition's findings, noting that methods relying on history alone "fail to effectively account for the effect of holidays, special events, promotions, prices and possibly the weather." It is also the finding most often skipped, because those variables live in another team's system.

An honest complication: to use a future promotion as an input, you must know the future promotion. If the promotional calendar is not fixed when you forecast, you have swapped a demand-forecasting problem for a promotion-forecasting problem, which may be harder. Two workable responses: restrict yourself to drivers genuinely known in advance (holidays, paydays, fiscal-year boundaries), or produce the forecast conditional on a stated plan and label it so, meaning that when the plan changes everyone knows the forecast changed with it.

One more distortion is invisible until someone points at it. If you sold out, your records show what you shipped, not what was wanted. Censored demand looks like low demand. The model learns to forecast low, the system orders less, and the next stockout arrives sooner. Any series where supply constrains the recorded number needs stockout periods flagged, or the forecast becomes a machine for reinforcing its own error.

Intermittent demand, and where the average lies to you

Go granular enough and most demand series are mostly zeros with occasional spikes: specialty parts, low-turn medications, rare configurations. Classical methods for this remain relevant: Croston's method and its variants split the series into the size of a demand when it occurs and the interval between occurrences, then forecast those two things separately.

The M5 results contain the best single illustration we know of for why aggregate accuracy misleads. Comparing Croston's method against seasonal naive across the competition's twelve aggregation levels, the paper reports seasonal naive was on average 11.5% more accurate — but the improvements were "extremely uneven." They start at 37.8% at the highest aggregation level, drop to 9.7% at level 9, and go negative at levels 10, 11 and 12, where Croston was more accurate than seasonal naive by 13.0%, 20.2% and 27.0% respectively. Level 12 is product-store, the most granular level in the dataset.

Read that as a management lesson rather than a methods lesson. The headline number says one method wins; the level where the ordering decision is actually made says the opposite, by a wide margin. A single overall accuracy figure is capable of pointing an organisation at exactly the wrong choice. Report accuracy by segment — volume tier, intermittency, newness, site — or you are reporting an average of dissimilar things.

Pick the metric and the backtest before you look at results

Choose the metric first, in writing, before anyone has seen a result. That removes the option of picking the measure that flatters the model you built.

Avoid MAPE on anything with small or zero values. Forecasting: Principles and Practice states the problem directly: percentage errors are "infinite or undefined" when the actual value is zero, and "put a heavier penalty on negative errors than on positive errors." That asymmetry quietly rewards models that under-forecast — the wrong incentive when a stockout is expensive.

Prefer scaled errors. MASE, proposed by Hyndman and Koehler, divides your error by the error a naive or seasonal naive forecast would have made on that series. Below 1 means you beat the baseline; above 1 means you did not. It is unitless, comparable across series, and puts the baseline comparison inside the metric rather than on a slide someone drops.

Backtest on a rolling origin. The procedure uses a series of test sets where "the corresponding training set consists only of observations that occurred prior to the observation that forms the test set," which is why it is also called evaluation on a rolling forecasting origin. Random k-fold splitting trains on the future and reports a number you cannot reproduce in production. Evaluate at the horizon you actually use: a model excellent one step ahead can be useless at the eight-week lead time your procurement cycle requires.

What to build first

None of this argues against modern methods. In the M4 competition, 12 of the 17 most accurate methods were combinations of mostly statistical approaches, and the six pure machine learning entries performed poorly, with none beating the combination benchmark and only one beating Naive2. In M5, on richer data, every top method was machine learning, most built on LightGBM, and the top 50 teams beat the best benchmark by more than 14%, with 22.4% for the winner. Bojer and Meldgaard's review of six Kaggle forecasting competitions found first-place solutions improving more than 25% over naive benchmarks. Foundation models are real too: the Chronos paper claims "comparable and occasionally superior zero-shot performance on new datasets, relative to methods that were trained specifically on them" — useful when you have thousands of series and no time to tune each one.

Complexity pays when the data supports it — grouped correlated series, real exogenous drivers, enough history — and not when it does not. You cannot tell which case you are in from a benchmark table built on someone else's data. So the sequence is:

1. Write the event log. What changed and when, before any modelling.
2. Separate zero, unobserved and not-yet-existing in the data, with a flag that survives into training.
3. Build seasonal naive plus one simple statistical method. Backtest rolling-origin. Report MASE by segment. That is your scoreboard, permanently.
4. Only then try a global model trained across series. Both M4 and M5 point to cross-learning and combining methods as reliable gains.
5. Ship intervals rather than point forecasts, then check whether the 90% interval contains the outcome 90% of the time. An unchecked interval is decoration.
6. Measure the decision, not the forecast. If the reorder point does not move, accuracy improved and nothing else did.

Most of the work described here is data work and evaluation design. That is where the time goes, and the part vendors have least incentive to describe.

Sources: Makridakis, Spiliotis & Assimakopoulos, "M5 accuracy competition: Results, findings, and conclusions," International Journal of Forecasting (participation counts, benchmark comparisons, the Croston vs. seasonal naive breakdown by aggregation level, and the validation-selection finding; a freely available preprint carries the same figures); Makridakis, Spiliotis & Assimakopoulos, "The M4 Competition: Results, findings, conclusion and way forward," International Journal of Forecasting (abstract); Bojer & Meldgaard, "Learnings from Kaggle's Forecasting Competitions"; Hyndman & Athanasopoulos, Forecasting: Principles and Practice (3rd ed.) on simple benchmark methods, forecast accuracy and MASE, time series cross-validation, and missing values and outliers; Prophet documentation on trend changepoints; Nixtla StatsForecast documentation on Croston's method; Ansari et al., "Chronos: Learning the Language of Time Series". Analysis and framing by Precision AI Academy.

Common questions

Why should I compare a forecasting model against a naive baseline? Because the baseline is the only thing that tells you whether the model added anything. In the M5 Accuracy competition, 5,507 teams competed on real retail demand data; per the published results, 2,666 teams (48.4%) beat the Naive benchmark, 1,972 (35.8%) beat the seasonal Naive benchmark, and only 415 (7.5%) beat the best statistical benchmark the organizers supplied. A model that is not scored against a baseline has not been evaluated at all.

Do complex models beat simple ones for demand forecasting? It depends on the data, which is exactly why you measure. In the M4 competition the six pure machine learning entries performed poorly, with none beating the combination benchmark and only one beating Naive2. In the M5 competition, which used grouped correlated retail series with prices, promotions and events, the top methods were machine learning built on LightGBM, and the winner beat the best benchmark by 22.4%. Same field, opposite outcomes, different data.

How do I handle missing periods in a demand history? Distinguish three cases that look identical in a table: true zero demand, a recording gap where demand may have occurred but was not captured, and a period before the item or site existed. Store an observed flag alongside the value, mask unobserved periods out of the training loss rather than filling them with zero, and never impute using values from after the forecast origin, which leaks future information into the backtest.

Which error metric should I use for intermittent demand? Not MAPE. Per Forecasting: Principles and Practice, percentage errors are infinite or undefined when the actual value is zero and put a heavier penalty on negative errors than positive ones, which quietly rewards under-forecasting. Scaled errors such as MASE, proposed by Hyndman and Koehler, compare the model against naive or seasonal naive forecasts on the same series, so a value below 1 means you beat the baseline.