Hit enter after type your search item

A first post-mortem

/
/
/
543 Views

Based on the latest results, Donald Trump won the election with 305 electoral votes (vs. 233 for Hillary Clinton). Most likely, the final outcome will show a split between the Electoral College and the national popular vote. The New York Times currently projects that Clinton will win the popular vote by 1.3 percentage points.

These results are a major upset for the PollyVote and other election forecasters, as virtually no one saw this coming.

Electoral College

The Electoral College outcome came at a big surprise. The PollyVote’s final Electoral College forecast predicted Clinton to win 323 electoral votes, compared to 215 for Trump, which was very much in line with other forecasters.

For these state-level forecasts, the PollyVote combined predictions from eight poll aggregators, five models, four sources of expert judgment, two prediction markets, and one source of citizen forecasts.

Not a single one of these twenty different sources predicted that Trump would win a majority of the electoral votes! Accordingly, the combined forecast failed as well.

Popular Vote

The final PollyVote forecast predicted Clinton to win the popular vote by 5 percentage points, 52.5% vs. 47.5%. Based on the latest projections, Clinton is expected to win the popular vote by about 1.2 points, which would translate to about 50.6% of the two-party vote. If these numbers are correct, the PollyVote will have missed by 1.9 percentage points, which is more than three times the average error from previous elections.

However, the accuracy of forecasts shouldn’t just be judged based on final predictions. Rather, one should look at which forecasting method has performed best over the course of the campaign.

The PollyGraph below shows the mean absolute error for the PollyVote and each of its component methods, across the remaining days to election. That is, at each day, the PollyGraph shows the error that you would have received by relying on that method’s forecast until Election Day. For example, starting March 15th, which is the day from which forecasts from all six component methods were available, the PollyVote’s average error until Election Day was 2.3 percentage points. This is makes it rank third after citizen forecasts (MAE: 1.1 percentage points) and and econometric models (MAE: 1.8 percentage points). Prediction markets were least accurate and missed on average by 5.9 points. (Note that these numbers will still change as the final vote tally changes. The chart below will always show the latest figures.)

NOTE: THE CHART IS CURRENTLY BROKEN

The relative accuracy of the different methods changes as we get closer to the election. For example, the average of econometric models became more accurate, whereas the accuracy of citizen forecasts decreased somewhat.

Initial thoughts

It’s been a long night and it’s still too early to draw conclusions. But Polly the forecasting parrot feels miserable about her miss. While she did of course perform as well as the typical forecast – which is the minimum what you would expect from a combined forecast – she did not outperform the best individual method as in previous elections. We will work hard to find out what went wrong and what information we missed in order to provide her with even better information next time.

But a look at the method’s relative accuracy in predicting the vote shares provides some interesting first insights. Prediction markets, which were among the most accurate methods historically, experienced a huge error. In contrast, econometric models, which were among the least accurate methods over the last six elections, outperformed all other methods in 2016.

Yesterday, probably no one – us included – would have thought that the econometric models component would turn out to be most accurate. Some forecasters didn’t even trust their own models. Others used the models’ forecasts only as a benchmark to estimate how many votes Trump would cost the Republican party.

These results conform to what we found in prior research on combining forecasts:

  1. The accuracy of different methods changes over time and across elections. There is simply no one best method to predict elections. The methods’ relative accuracy for forecasting an election strongly depends on the context and the idiosyncrasies of that particular election.
  2. It’s very difficult to determine a priori which method will provide the best forecasts. This is also why it makes little sense to weight forecasts based on their historical accuracy, particularly in applications such as election forecasting, where we have limited historical data to derive valid estimates about the method’s relative performance.

This is why Polly’s method of combining different forecasts is still valuable, and perhaps even more important in the future. First, it’s generally not a good idea to trust any single forecast. Second, in aggregating all the different forecasts that are out there, the PollyVote enables us to learn about the accuracy of the various methods – and the biases they may entail – over time.

If we can better understand the conditions under which different methods work best, we might be able to use this information to improve the accuracy of our combined forecast and to better estimate its surrounding uncertainty. In fact, estimating uncertainty is one of the most interesting areas for future research when it comes to combining forecasts.

We will provide additional analyses and thoughts over the next days and weeks. So stay tuned.

This div height required for enabling the sticky sidebar