As AI and machine learning continue to transform demand forecasting, one question remains: what is still the role of the planner? While statistical forecasting or machine learning models provide a solid baseline for most products, planners continue to enrich forecasts with market intelligence, customer information, promotions, or other business insights. These enrichments can add significant value, but they can also reduce forecast accuracy. So how can we tell the difference?
During her master’s thesis at EyeOn, Milou Wintjes investigated whether AI can help answer this question. Using machine learning, she explored whether the characteristics of forecast enrichments, together with the written rationale behind them, can predict whether an adjustment will ultimately improve forecast accuracy.
The challenge of judgmental forecasting
Most forecasting processes combine statistical or machine learning forecasts with human expertise. Planners adjust system-generated forecasts to incorporate information that is not captured by the model. While these enrichments can add significant value, not every adjustment improves the forecast. The process usually follows the following steps; a planner cleans historical data, the system creates a forecast, this is reviewed by the salespeople and the demand planner, and they make adjustments. This results in a final forecast, as shown in the figure below.
The set-up of this process raises an important question for organizations:
How can planners distinguish valuable enrichments from those that unintentionally harm forecast quality?
Looking beyond the forecast adjustment itself
Most Forecast Value Added (FVA) analyses focus on the adjustment itself:
- How large was the adjustment?
- Was the forecast adjusted up or down?
- Did forecast accuracy improve after making the adjustment?
Milou’s research went one step further. In addition to analyzing adjustment characteristics, she investigated whether the written explanations behind enrichments contain signals that help explain forecast performance.
Applying machine learning
Milou collected data from one of our customers. With these adjustments and reasons, she evaluated the forecast value add of the adjustments.
A Random Forest machine learning (ML) model was then trained to identify which enrichments were likely to improve forecast value add. This ML model included two groups of features:
- Adjustment features (direction and size)
- Decision rationale features extracted from planner comments, such as word count, comparison with a previous period or product.
An example of an adjustment can be found below:
The results were clear: the strongest predictors of forecast value add were:
- Adjustment direction
- Adjustment size
In particular, downward adjustments were far more likely to improve the forecast than upward adjustments, with larger downward adjustments delivering the highest value. Small upward adjustments provided the least value.
One of the most surprising findings was that the written explanations behind enrichments had only a limited impact on the model’s predictions. The strongest signals came from the enrichments themselves, with adjustment direction and size proving far more predictive than the underlying comments.
Does this mean comments are unnecessary?
Absolutely not. Comments remain highly valuable because they:
- Increase transparency
- Support learning and continuous improvement
- Explain the reasoning behind adjustments
The limited impact of comments in this study is likely related to the quality and consistency of the available data. Many comments were short and lacked detail, making it difficult for the model to extract meaningful insights. With richer and more structured comments, we expect their value to increase significantly.
From insights to action
The ultimate goal was not only to understand forecast enrichments, but also to improve the forecasting process. Based on the model predictions, Milou developed a policy that evaluates whether an enrichment should be:
- Accepted
- Rejected and replaced by the statistical forecast
The machine learning model was even translated into a simple and practical decision rule which is actively used by the demand planning team:
What does this mean for other organizations?
The findings offer several valuable lessons for forecasting teams:
- Human judgement remains important and often adds value.
- Adjustment direction and size are the strongest predictors of forecast value add.
- Machine learning can help identify potentially harmful enrichments before they impact operations.
- Structured documentation on the reasoning behind adjustments remains important for accountability, learning, and future improvement.
Optimal use of human judgement in times of AI
Milou’s thesis highlights an exciting development within supply chain planning: the combination of human judgment and AI.
By understanding not only what planners adjust, but also when adjustments are most likely to add value, organizations can develop smarter forecasting processes that make better use of both human expertise and machine intelligence.
At EyeOn, we are proud to support pioneering research like this and help organizations translate these insights into practical improvements in forecasting and planning performance.
Would you like to learn more about forecast value add, judgmental forecasting, or how AI/ML can support your planning process? Feel free to get in touch with our experts Bregje van der Staak or Caitlin Riesewijk.