Öberg: The Riksbank's forecasting performance
Introduction
”When one remembers how foolishly one thought and how wrong one was and how different things have usually been and how the world has been affected by strange metamorphoses, one has less and less respect every year for forecasts.” This is a fairly well-used quotation from Alf Henriksson. And they are wise words. But at the Riksbank we have no choice, as the interest rate affects inflation and the rest of the economy with some time lag. Our decisions must therefore be based on forecasts. The question I wish to discuss today concerns our forecasting performance.
Evaluation is important for independent central banks
Experiences show that the work on maintaining low and stable inflation is made easier if the central bank can act independently. At the same time, greater independence makes demands on the central bank to be open and to clearly explain its decisions. There are several reasons why this is important.
One important aspect is that openness makes it easier to evaluate and to require accountability, which is essential for a central bank’s legitimacy. Openness enables the general public to understand what the central bank is doing. This in turn makes demands on the central bank to be clear in its analysis and message. While the openness makes it easier to evaluate monetary policy, one can say that the evaluations are also an important part of the openness.
Only a couple of decades ago central banks were fairly secretive. Very little of what was carried out became known to the general public. There have been considerable changes since then. The process of development from uncommunicativeness to increasing openness that has taken place at central banks around the world has probably contributed to the success of monetary policy. What I mean here is that they have succeeded in anchoring inflation expectations at a low and stable level.
The Riksbank is one of the most open central banks of all. We began publishing our inflation forecasts in 1997 and since then we have gradually become more open in our view of economic developments. Other central banks are also moving towards greater openness. For instance, only a few days ago the Federal Reserve in the United States began publishing forecasts four times a year instead of twice. They have also extended the forecast horizon from two to three years, just as we at the Riksbank did a couple of years ago.
It is thus essential that monetary policy is evaluated and discussed. This applies both to the principles behind monetary policy and to how well we succeed in meeting our objective. To make this possible there must be a clear basis for the decision. It must be clearly shown which principles we have based our monetary policy decisions on and what information our conclusions are based on.
As monetary policy affects the economy with some time lag, the monetary policy decisions must be based on forecasts, as I mentioned at the start. Good bases for decisions in the form of accurate forecasts are thus an important factor in the success of monetary policy. But future developments in inflation and other variables are uncertain. For one thing, there is the uncertainty linked to the statistical methods and models, as certain fundamental parameters vary over time. For another thing, there are a number of other factors that can affect inflation in a way that is very difficult to predict. For instance, increased geopolitical unease can push up the oil prices, or an unexpected outbreak of a disease among animals can lead to a rapid rise in food prices.
When central banks make forecasts, it is therefore increasingly common that they put a band around the forecasts that measures their uncertainty. These bands help provide information about the uncertainty in the forecasts and also make it easier to communicate the monetary policy alternatives that may arise in the future.
But what do we know about the forecasting performance of the Riksbank?
Forecasting performance is evaluated both internally and externally
Forecasting performance can be evaluated in many different ways. There are a number of statistical methods that can be used to analyse how accurate the forecasts are. One means of measuring accuracy is to compare the forecasts with outcomes. The problem is that one doesn’t know whether a forecasting error is due to the forecasting method being inadequate, or to something genuinely unpredictable. It is therefore common to compare accuracy with forecasts from simple time series models to gain a perspective on whether the forecasting errors are unusually large or small.
We have made a number of evaluations of our own forecasting performance and we have also been evaluated by external experts. In recent years, we ourselves evaluated some of the models used in the forecasting work. Various types of model are used in the forecasting work to produce the best possible base for an assessment of economic developments. A study that will soon be published has investigated the forecasting precision of our general equilibrium model and the Bayesian VAR model(1). Another study has evaluated our indicator models(2).
In addition we publish information to enable an assessment of the monetary policy conducted over the past 2-3 years in a special appendix to the first Monetary Policy Report of every year. It contains, for instance, comparisons of our forecasts with other forecasts and with outcomes. The material we produce in our annual evaluations forms an important basis for the Riksdag Committee on Finance’s report on monetary policy.
Every year the Committee evaluates at its own initiative the monetary policy conducted during the previous years. They have done so since 1999 when the Riksbank became independent. Last year we also had two external researchers, Francesco Giavazzi and Frederic Mishkin, who made a more in-depth evaluation of monetary policy. This was commissioned by the Committee on Finance and covered the entire ten-year period of inflation targeting.
New study of the Riksbank’s forecasting performance
As a step in the evaluation work, three economists at the Monetary Policy Department of the Riksbank recently made a comprehensive analysis of our forecasting performance during the period 2000-2006(3). They have analysed the forecasts of the most important variables for monetary policy – GDP, employment, productivity and inflation(4).
In the study they have tried to estimate how accurate the forecasts have been on average by comparing them with outcomes. To gain a perspective of the accuracy, our forecasts have also been compared with forecasts from a simple time series model. The so-called AR models, which are often used as a standard comparison in evaluations of this type, quite simply function so that one adjusts the variable in question with the aid of its own historical data. The type of time series model chosen for the comparison is, despite its simplicity, difficult to surpass, according to several international studies.
Moreover, they have analysed how far ahead we can make forecasts with good accuracy, that is to say what forecasting memory the different variables have(5). They have also looked at whether there are tendencies to systematically over-estimate or under-estimate the different variables, that is, whether the forecasts have any bias.
What does the analysis show about our forecasting performance?
Let us first look at the entire period and compare our forecasts for GDP, employment and CPIX with outcomes. CPIX is a measure of underlying inflation, which until recently was known as UND1X. Hardly surprisingly, it appears that the forecasts have varied in accuracy (Figure 1). For instance, we have under-estimated employment growth recently (Figure 2). Nor did we predict the sharp rise in inflation spring 2001 in our assessments the previous year. However, we were more successful in capturing the downswing in inflation during 2003 (Figure 3). But how do things look in more detail for the different variables?
Forecasts for GDP growth
With regard to the GDP forecasts, we have periodically been accurate and close to the outcome, while we have fallen short on other occasions (Figure 1). It is often particularly difficult to make forecasts in an economic turnaround.
Compared with the time series model that is used to gain a perspective of the forecasting memory, we make better GDP forecasts up to a year ahead. With regard to the forecasting memory for GDP, that is, how far ahead we can make a forecast with some information value, this stretches four quarters ahead (Figure 5). This agrees with experiences in other countries. One result of an evaluation that analysed the forecasting memory of Canadian and US macro variables shows that it is often limited to the first forecasting year(6).
Another means of analysing the forecasting performance is to look at the correlation between forecast and outcome (Figure 9). The correlation shows the strength in the relationship between forecasts and outcomes. A correlation of one year is a perfect relationship, if it is zero there is no relationship at all, while minus one is a perfect negative relationship. The correlations shall thus preferably be positive and as close to one as possible.
One year ahead the correlation between the GDP forecast and outcome is just over 0.6 and two years ahead it is around 0.1. This supports the picture we obtained when we compared out forecasting memory with the time series models. We were better than the model up to one year ahead and we had a forecasting memory four quarters ahead.
Forecasts for employment
If we instead look at employment forecasts, we believed that employment would rise more in 2003 and 2004 than it actually did (Figure 2). Instead, productivity growth was repeatedly stronger than expected. But on the other hand, employment has been stronger than expected in our assessments over the past year. The forecasts we have made for employment growth are on the whole more accurate than those of the time series model and the forecasting memory stretches eight quarters ahead (Figure 6).
If, on the other hand, we only look at developments over the past two years, the Riksbank has succeeded in forecasting employment about as well as the time series model. This means that we have not succeeded in adding any further information on top of what the simple model can capture.
How is the relationship between employment forecasts and outcome? The correlation is almost 0.9 one year ahead and just over 0.4 two years ahead (see Figure 9). The correlation is stronger than for GDP and agrees with the picture we obtained when we compared the forecasts with the time series model. The employment forecasts appear on average to have been more accurate than the GDP forecasts.
Forecasts for productivity
Productivity is a central variable in the monetary policy analysis. The most common measure of productivity is labour productivity, that is, the ratio between GDP and the number of hours worked. It has not been possible to evaluate this measure here, as there is not enough historical forecasting data on the number of hours worked. Instead the ratio between GDP and employment has been used as an approximation.
The analysis shows that our forecasts have under-estimated productivity growth on all forecasting horizons (Figure 4)(7). The time series model is on the whole slightly more accurate than our forecasts have been during this period. The forecasting memory for productivity has only extended one quarter ahead (Figure 7). The same applies to the correlation between forecasts and outcomes. It is 0.7 one quarter ahead, while it is zero both one and two years ahead (see Figure 9).
Inflation forecasts
When it comes to inflation measured as the annual rate of increase in the CPIX, we have made fairly accurate forecasts during a large part of the period evaluated, that is, 2000-2006 (Figure 3). Something we missed was the sharp upturn in inflation in spring 2001. This was primarily due to an outbreak of mad cow disease which led to rising food prices. But since then we have made inflation forecasts that have been close to the outcomes. For instance, the downswing in inflation in 2003 was captured to a greater extent than the upswing at the beginning of 2001. The inflation forecasts are on the whole more accurate than the time series models. The forecasting memory for the CPIX extends eight quarters ahead (Figure 8).
The correlation between forecasts and outcome is almost 0.8 one year ahead and just over 0.5 two years ahead (see Figure 9). None of the other variables has as high a correlation two years ahead. The fact that the correlation wanes in time is natural as the uncertainty in the forecasts increases.
Is there any bias one or two years ahead in the forecasts?
Let us now look at whether there is any bias, that is, systematic over-estimation or under-estimation in the forecasts one and two years ahead. The smaller the bias a forecast has, the better it is.
One year ahead the bias for annual GDP growth is very small. However, two years ahead the forecasts have been almost been four tenths too high on average, compared with the outcomes, which is not negligible (Figure 10).
With regard to the annual development in employment, the bias is just under three tenths one year ahead and two tenths two years ahead. Employment has been under-estimated in both cases, that is, the outcomes have on average been higher than in the forecasts.
Productivity has also been under-estimated both one and two years ahead. Productivity has a bias of around one tenth one year ahead and just under seven tenths two years ahead. We have thus made a relatively large under-estimation of productivity growth two years ahead(8).
With regard to the annual change in inflation measured as the CPIX, the forecasts have on average been almost three tenths lower than the outcomes one year ahead. However, the forecasts two years ahead have on average been two tenths higher than the outcomes, which can be regarded as a minor deviation.
What conclusions can we draw from this? The bias is relatively small for most of the variables. There are no clear signs of any systematic errors in the forecasts, with the possible exception of productivity. The rate of increase in productivity has been systematically under-estimated during the period. However, it should be added that none of the variables has any statistically significant bias.
Comparison with other forecasters
The study has also compared the Riksbank’s forecasting errors with the National Institute of Economic Research (NIER) and Consensus Forecasts panel, a compilation of forecasts from a number of different forecasters. It is clear from the study that the Riksbank’s forecasts for GDP growth have been among the best compared with other participants in the Consensus panel (Figure 11). On around half of the occasions we have made more accurate forecasts than 80 per cent of the other forecasters. The inflation forecasts measured as the annual rate of change in the CPI have in almost 75 per cent of cases been more accurate than half of the forecasters in Consensus panel (Figure 12). In a comparison of quarterly GDP and CPIX forecasts, it is shown that the Riksbank has on the whole made slightly more accurate forecasts than the NIER. At the same time, it should be said that the differences in forecasting performance are not statistically significant.
If we compare the results with an earlier forecast evaluation made by the Riksbank, there are some similarities. In this evaluation we found, for instance, that the NIER has made better forecasts of productivity than we have, at the same time as we have made better forecasts of CPIX inflation(9).
There are of course a number of difficulties in making correct comparisons of this nature, which is also indicated in the study. It is important to note that the Riksbank made forecasts under the assumption of an unchanged repo rate up to the publication of the second Inflation Report of 2005. With this method CPI inflation should actually be over-estimated in the short-term if the interest rate is cut. It is actually only with effect from this year, when we have begun to make forecasts based on our own interest rate path, that the forecasts fully mirror our views of the future.
Another point is that the forecasts are rarely made at the same point in time, which can of course make accuracy more difficult. However, the analysis here is designed to ensure this problem is as small as possible.
Forecasts are uncertain and we mark this by an uncertainty band
Forecasts are uncertain and one is not always right. Most of those who make forecasts are very well aware of this. Back in 1997 when we began publishing forecasts of inflation we reported uncertainty bands around the forecasts. As uncertainty increases over time, the uncertainty band takes the form of a fan (Figure 13). We now also publish uncertainty bands for the GDP and repo rate forecasts.
The uncertainty that is reflected in the bands can be divided into three parts. It reflects the fact that we are uncertain about the current situation, regarding how the economy functions, and about what shocks might occur during the forecast period. The uncertainty bands are constructed so they are symmetrical around our main scenario and for GDP growth and the inflation rate the breadth of the bands is calculated with the aid of our earlier forecasting errors. This gives an incentive to make good forecasts, as large forecasting errors bear the penalty of broad uncertainty bands and the risk of reduced credibility in the forecasts.
The uncertainty band for the repo rate cannot be calculated in the same way, as we have only begun to make our own forecasts for the repo rate this year. There are quite simply insufficient historical forecasts to use as a base. The uncertainty band for the repo rate has instead been constructed with the aid of forecasting errors among the so-called implied forward rates (Figure 14). The implied forward rates reflect market expectations of the repo rate. One problem with this method is that the implied forward rates have systematically over-estimated the future repo rate path for a number of years. However, the over-estimation has been removed from the forecasting errors, with reference to the fact that the forward rates also consist of a risk premium that should not be included when using them for forecasts.
Different central banks have chosen different methods of illustrating the uncertainty in the forecasts and we regularly compare notes with one another.
We are continuing to develop our forecasts
The conclusion from this review is that the Riksbank’s forecasts on the whole are rather good, particularly one year ahead.
We have made good forecasts for all variables, with the exception of productivity, one year ahead. This is reflected in the fact that we have made more accurate forecasts than the time series model and also in the fact that the correlations are relatively high between forecasts and outcomes. In addition, the bias is small for all variables one year ahead.
We have a forecasting memory two years ahead for both employment and inflation. Although the correlations between forecasts and outcomes are weak one year ahead, they nevertheless show a clear relationship. On the other hand, the study implies poor forecasting memory for GDP and productivity two years ahead. The correlations between the forecasts and outcomes are also weak for these variables two years ahead. We have seen that the bias is larger for most variables two years ahead. GDP and inflation have been over-estimated slightly during the period studied, and employment and productivity have been under-estimated.
Our inflation forecasts, particularly those for CPIX, have good qualities such as high correlation with outcomes and long forecasting memories. It is worth pointing out that this is the variable that is most important for monetary policy. At the same time, the uncertainty band shows that probability that inflation will be within the Riksbank’s tolerance interval of 1-3 per cent two years ahead is only around 75 per cent, despite our forecast being that it will be two per cent.
On the other hand, our forecasts for the real economy are not as accurate as the inflation forecasts, although they compare fairly well with the time series model’s
forecasts.
Our forecasts also compare fairly well with those of other forecasters. The result of the comparison with other forecasters shows that our GDP forecasts up to one year ahead are often among the most accurate. This also applies to the CPI, but here the difference is smaller compared with the NIER and the Consensus panel.
This does not mean that we have not occasionally made considerable forecasting errors. But the errors are not so systematic that there is reason to believe they are due to inadequate forecasting methods. And nor does this mean that we cannot or do not wish to become even better. We are constantly working to improve our methods. In some areas we need to work extra hard to improve our forecasting performance. One such area is productivity.
It is important that we continue to regularly assess monetary policy, and particularly with regard to our forecasts, both in our annual assessments and in other ways. This is necessary both to be able to gain better knowledge of which variables are difficult to forecast and to see where it is necessary to develop methods and analysis. A major advantage of publishing our forecasts is that external experts can also evaluate us.
Thank you for inviting me here today!
Notes:
1. “Modern forecasting models in action: Improving analyses at central banks” Adolfson, M., M. K. Andersson, J. Lindé, M. Villani & A. Vredin, (2007), International Journal of Central Banking, in the process of being published.
2. ”The Riksbank’s new indicator procedures” Andersson, M. and M. Löf, Sveriges Riksbank Economic Review 2007:1, Sveriges Riksbank.
3. Andersson M. K., Svensson J. Karlsson G., “An evaluation of the Riksbank’s forecasting performance”, Sveriges Riksbank Economic Review (2007:3), Sveriges Riksbank.
4. The forecasts for employment and productivity have been evaluated during the period 2002-2006 as there are no quarterly forecasts documented further back in time for these variables.
5. The forecasting memory is defined as the longest horizon where the forecast error variation (measured as the root mean square error) is lower than the series’ variation (standard deviation).
6. See, for example, Galbraith, J. and G. Tkacz, “How far can we forecast?: Forecast content horizons for some important macroeconomic time series”, Department of Economics Working Paper 13 (2006), McGill University.
7. We have earlier analysed this in the context of a structural model, Ramses. The evaluation appendix to Inflation Monetary Policy Report 2007:1 contains an analysis of, for instance, the effects of productivity shocks on the economy.
8. During the evaluation period for productivity, which extends from 2002 to 2006, we have under-estimated both GDP and employment in our forecasts. Please note that the bias estimate for GDP changes character depending on whether the sample refers to 2000-2006 or 2002-2006.
9. This result from an earlier forecast evaluation applies only to forecasts for the next year. The result is statistically reliable.
