When creating a forecast, one question always comes up: how can I achieve optimal forecast accuracy?
Several factors go into creating an accurate forecast, including the appropriate algorithm selection, parameter choices, data selection, error measurement, and more... unless you are using some black box model!
The system you are using can help you with some of these choices: For example in SAP Integrated Business Planning we have quite a few algorithms that provide automated parameter selection and we provide vast documentation on the strengths and weaknesses of the different algorithms.
However, there is one point that is often overlooked. Once you've decided on a set of algorithms to experiment with and chosen the parameters (or you let the system chose them for you), and you've identified the most fitting error measure for your requirements, the next question you should be asking is: which data will be used to test the quality of the different algorithms?
There are two alternatives: either use the ExPost forecast or a Test Phase.
As these terms might be unfamiliar to some, let's begin with the basics.
What is ExPost forecast?
ExPost forecast is a prediction calculated for past periods for which actual demand history is also available. It is used to assess the accuracy and effectiveness of forecasting models by comparing model predictions made in the past with the actual outcomes that we now have data for. Essentially, it provides a measure of model performance. The term "ExPost" is Latin for "after the event".
Here is a very simple example of ExPost using simple moving average as the forecasting method:

10 
9 
8 
7 
6 
5 
4 
3 
2 
1 
Current Week 
+1 
+2 
+3 
Sales history 
20 
30 
40 
20 
21 
32 
30 
40 
20 
21 




ExPost Forecast 



30 =(20+30+40)/3 
30 = (30+40+20)/3 
27 = (40+20+21)/3 
24 = (20+21+32)/3 
28 = (21+32+30)/3 
34 = (32+30+40)/3 
30 =(30+40+20)/3 




Forecast 










27=(40+20+21)/3 
27 
27 
27 
ExPost forecast can be calculated differently depending on the algorithm. In particular, each algorithm can have different “warmup” periods (in the example with Moving Average above, the first three periods are such warmup periods).
A simple example of ExPost forecast of regression algorithms
Following is a simple example of how ExPost forecast is computed in case of regression algorithms (e.g. the Machine Learning algorithm Gradient Boosting of Decision Trees). In case you are not familiar at all with regression algorithms, you can skip this section.
In case of regression algorithms, the model learns from past correlation of the sales history and additional data. And then applies the learning to the past periods.
Let’s assume you have trained some kind of decision tree model on your sales history using events, promotions and the month of the year (this one is systemgenerated in SAP IBP) as additional data. Here is an example of the sales:
Month of the year 
1 
2 
3 
4 
5 
6 
7 
8 
9 
10 
11 
12 
1 
2 
Sales history 
20 
30 
40 
20 
21 
32 
30 
40 
20 
21 
25 
20 
21 
30 
Event 





X 
X 







Promotions 


10 


10 

10 






The algorithm then might have learned this rule:
This rule is based on observations of the past sales history. Now the decision tree that the algorithm created using the sales history will be used to create the ExPost forecast.
The ExPost forecast for month 6 will be exactly 32 because the algorithm has learned that if promotions is 10 and there is an event, then the sales are equal to 32. Notice that if you had been in month 5 you could not have had this information at that time! As you can see, the ExPost forecast uses past sales to predict past sales, even if at that point of time the actual sales were not known!
The ExPost forecast is
not the same as an offset forecast, where you pretend to be in the past and look at how the forecast “would have looked like”. It simply gives an indication of how well the algorithm approximates the sales history.
Since the expectation is that the future sales will resemble the past sales, ExPost forecast is still a valid tool to measure forecast accuracy. The system calculates the forecast accuracy by comparing the differences between the actual values and the ExPost values.
Various forecast algorithms may have varying initialization periods, i.e., they can begin generating ExPost forecasts in different time buckets. This variability can affect the error calculation for the ExPost forecast. In SAP IBP, there are several strategies to approach this issue. One such method  Error Measure Calculation Horizon  can be found on the Forecast Model's General tab. This function allows the user to specify how many recent historical periods should be factored into error measurement calculations. This ensures that the error is assessed based on the same number of time periods, irrespective of the algorithm used.
What is a Test Phase?
In a statistical context, a Test Phase refers to a specific period set aside in the model development process to assess the performance of a statistical model or algorithm. During this phase, the model is applied to a test data set a subset of the original data that was not used during the model training. The model's predictions are then compared against the actual results in the test data. This provides meaningful insights into how well the model or algorithm is likely to perform on new, unseen data.
The Test Phase always consists of the most recent periods in the historical horizon. For example, if the current month is January and the number of Test Phase periods is 4, then the Test Phase consists of the last 4 months of the previous year, from September to December.
When a Test Phase is applied, each algorithm that you chose to compare calculates a forecast for the Test Phase based on historical data from the periods before the Test Phase. The chart shows this process.
In SAP IBP you can define a Test Phase by maintaining the Test Phase Periods field on the Forecasting Steps tab of the Forecast Model. The system will preserve this number of the latest historical time periods for testing purpose, and let each algorithm calculate forecast for this period from the remainder of the history horizon. The calculated forecasts are compared with the actuals, and the most accurate algorithm is selected.
Conclusion  How to choose the best algorithm?
Now that we explored how the accuracy of different algorithms is measured within a Best Fit with both ExPost and Test Phase, let’s see which one you should use.
Generally speaking, it’s best practice to use a Test Phase rather than ExPost forecast to determine the best algorithm in a Best Fit model.
This is because ExPost forecast can lead to overfitting of the algorithm, meaning that the algorithm is
very good at approximating the sales history. As a result, the forecast accuracy of the ExPost is very good, but the generated forecast still has poor quality.
Some algorithms have more accurate ExPost forecast then others, but it does not necessarily mean that their forecast will also be more accurate. For example, the exponential smoothing type of algorithms, especially in case of high smoothing coefficient, tend to calculate very accurate ExPost forecast.
Using a Test Phase ensures that you are checking the accuracy of the algorithm on a completely different set of data, therefore having more realistic view on which algorithm will perform best in the future. Keep in mind that unless you specify a Test Phase, SAP IBP uses ExPost forecast by default for the Best Fit.
Are there some cases in which ExPost still makes sense? The answer is yes. One very good reason to keep using ExPost forecast is the amount of data you have. To make a very simple example, you can’t use a Test Phase of 6 months if the total sales history is 8, as this would not leave enough datapoints for the algorithm to learn (especially if we are talking about more advanced algorithms that need a lot of data). If you are forecasting seasonal time series, then you should leave at least enough data to cover two season cycles.
Also note that if there is a change point (e.g. a significant level shift) in the test phase, the algorithm selection might not be optimal.
I hope this helped you in your journey to optimize your forecast. Do not hesitate to contact me for any questions regarding ExPost and Test Phase in SAP IBP for demand.
You can find more information on the topic of choosing the best algorithm in the official help: