Combining forecasts

The PollyVote is based on the principle of combining forecasts. That is, PollyVote combines forecasts from different forecasting methods, the so-called component methods, each of which relies on different data. The PollyVote forecast is calculated by following a two-step approach:

1. Averaging forecasts within each component method.
2. Averaging the resulting forecasts across component methods.

In other words, we use equal weighting of all forecasts within each component method, then equal weighting across these forecasts from different methods. This is the same approach that the PollyVote successfully used to forecast U.S. presidential elections since 2004, and the 2013 German federal election.

The rationale behind choosing this procedure is to equalize the impact of each component method, regardless of whether a component includes many forecasts or only a few. For example, while there is only one prediction market that predicts the national popular vote, there are forecasts from numerous econometric models. In such a situation, a simple average of all available forecasts would overrepresent models and underrepresent prediction markets, which we expect would harm the accuracy of the combined forecast. Another advantage of this approach is that it allows for comparing forecasts from different component methods.

The principle of combining forecasts

Combining forecasts is well established in the forecasting literature as a powerful method to reduce forecast error. The research interest in combining forecasts has increased since the publication of the frequently-cited paper by Bates and Granger almost half a century ago. Since then, hundreds of studies have demonstrated the benefits of combining forecasts in different fields.

Why combining forecasts works

One intuitive explanation as to why combining improves accuracy is that it enables forecasters to use more information, and to do so in an objective manner. Moreover, bias exists both in the selection of data and in the forecasting methods that are used. Often the bias is unique to the data and the method, so that when various methods using different data are combined to make a forecast, the biases tend to cancel out in the aggregate.

How to best combine forecasts

A widespread concern when combining forecasts is the question of how best to weight the components, and many scholars have proposed different methods for doing so. However, an early review of more than two hundred studies from different fields concluded that the question of how to combine forecasts does not seem to be critical to forecast accuracy. In fact, it was found that the simple average (i.e., assigning equal weights to components) often provides more accurate forecasts than complex approaches to estimating ‘‘optimal’’ combining procedures. Empirical research since then repeatedly confirmed these findings.

One reason for the strong performance of equal weights is the fact that the accuracy of the component forecasts varies over time and depends strongly on external effects. For example, when analyzing the predictive performance of six econometric models across the ten election from 1976 to 2012, one study found a negative correlation between a model’s past and future performance. In other words, models that were among the most accurate in a given election tended to be among the least accurate in the succeeding election. The reason is that every election is held in a different context and has its idiosyncrasies and, therefore methods that worked well in the past might not work well in the future. In such a situation, weighting the forecasts based on past performance is of course of limited value.

Another more technical reason is estimation error in the differential weights. In general, the simple average will be more accurate than estimated ‘‘optimal’’ weights if two conditions are met:

1. The combination is based on a large number of individual forecasts and
2. The optimal weights are close to equality.

The reason for this is that, in such situations, each forecast has a small weight, and the simple average provides an efficient tradeoff against the error that arises from the estimation of weights.

Conditions for combining forecasts

While combining is useful whenever more than one forecast for the outcome are available, the approach is particularly valuable if

1.  Many forecasts from evidence-based methods are available.
2. The forecasts draw upon different methods and data.
3. There is uncertainty about which method is most accurate.

These conditions perfectly apply to election forecasting. First,there are many evidence-based methods for predicting election outcomes. Second, these methods often rely on different data. Third, in most situations, it is difficult to determine a priori which method will provide the best forecast at a given time in an election cycle.