IBP ARIMA AIC vs. BIC

[OliverA41]

Hello,

I have a general question. In IBP I have the forecast Model ARIMA. One of the settings are to choose between AIC and BIC.

Can you explain me the difference between both and furthermore, in which situation you will use AIC and wwhen BIC

Regards

Oliver

[lev_degtyarov · Accepted solution]

Hello Oliver!

I hope you learnt a lot about demand forecasting from your recent post!

Nice to see your here again with "right" questions!

AIC is basically suitable for a situation where you don't necessarily think there's 'a model' so much as a bunch of effects of different sizes, and you're in a situation you want to get good prediction error. As such, as the sample size expands, the AIC choice of model expands as well, as smaller and smaller effects become relevant (in the sense that including them is on average better than excluding them).

BIC on the other hand basically assumes the model is in the candidate set and you want to find it.

BIC tends to hone in on one model as the number of observations grows, AIC really doesn't.

As a result, at large n, AIC tends to pick somewhat larger models than BIC. If you're trying to understand what the main drivers are, you might want something more like BIC.

Best Regards!

Lev

[OliverA41]

Hi Lev,

thanks for your comment. I think it is very helpful.

So it sounds for me, that with AIC/AICc and BIC, that the modell and the result can be assesed (random noise)

thanks

Oliver

[OliverA41]

Hi Lev,

thanks for the response.

I try to understand more and more about time series planning. 🙂

Your answer was helpfull. For me it sound, that if you use the Arima Modell, you select not the RMSE (it's my favorite) but you used the MSE (MSPE is not available). Correct ?

Regards

Oliver

[lev_degtyarov]

Hello Oliver,

Classic metrics and Information criteria are inter-related, but they represent different objectives in choosing the best model.

Keep in mind that a more complex non-linear model (like SARIMA), is typically able to fit better to past data. But the problem is that noise or random fluctuations in the training data can be "learned as feature" by the model.

These features, learned in training data set, are not applied to new unseen data and negatively impact model ability to generalize. If model overfits then it will not produce good forecasts for unseen future data.

Thus, classic metrics (RMSE\MSE\MAD\MAPE etc.) can measure forecast error but it disregards the "complexity" of the model. Optimizing for RMSE can give you accurate results but could lead to overly complex model that captures too much noise in the data (overfitting).

An alternative is to use what is called Information criteria (AIC\AICc\BIC etc.). Information criteria try to balance how good a model fits vs. model's complexity, e.g. to find simpler but still reasonably accurate model.

In general Information criteria can be written as: Information criteria = goodness of fit + penalty for model complexity.

Model complexity is typically the number of model parameters scaled by some factor to make it comparable to the goodness of fit metric.

Finally, coming back to your question "in which situation you will use AIC and when BIC?"

I recommend you to use AICc in most of cases. Not only me, but this is what is also used by time-series forecast researches & data scientists.

Regards,

Lev Degtyarov