This paper provides a non-systematic review of the progress of forecasting in social settings. It is aimed at someone outside the field of forecasting, wanting to appreciate the results of the M4 Competition by reading a survey paper to get informed about the state of the art of this discipline. It discusses the recorded improvements over time in forecast accuracy, the need to capture forecast uncertainty, and what can go wrong with predictions. Subsequently, the review distinguishes knowledge achieved over the last years into (i) what we know, (ii) what we are not sure about, and (iii) what we don’t know. In the first two areas, we explore the difference between explanation and prediction, the existence of an optimal model, the performance of machine learning methods on time series forecasting tasks, the difficulties with predicting non-stable environments, the performance of judgment, and the value-added of exogenous variables. The article concludes with the importance of (thin and) fat tails, the challenges and advances in causal inference, and the role of luck.
This post was originally published on this site