A first post-mortem

Based on the latest results, Donald Trump won the election with 305 electoral votes (vs. 233 for Hillary Clinton). Most likely, the final outcome will show a split between the Electoral College and the national popular vote. The New York Times currently projects that Clinton will win the popular vote by 1.3 percentage points.

These results are a major upset for the PollyVote and other election forecasters, as virtually no one saw this coming.

Electoral College

The Electoral College outcome came at a big surprise. The PollyVote’s final Electoral College forecast predicted Clinton to win 323 electoral votes, compared to 215 for Trump, which was very much in line with other forecasters.

For these state-level forecasts, the PollyVote combined predictions from eight poll aggregators, five models, four sources of expert judgment, two prediction markets, and one source of citizen forecasts.

Not a single one of these twenty different sources predicted that Trump would win a majority of the electoral votes! Accordingly, the combined forecast failed as well.

Popular Vote

The final PollyVote forecast predicted Clinton to win the popular vote by 5 percentage points, 52.5% vs. 47.5%. Based on the latest projections, Clinton is expected to win the popular vote by about 1.2 points, which would translate to about 50.6% of the two-party vote. If these numbers are correct, the PollyVote will have missed by 1.9 percentage points, which is more than three times the average error from previous elections.

However, the accuracy of forecasts shouldn’t just be judged based on final predictions. Rather, one should look at which forecasting method has performed best over the course of the campaign.

The PollyGraph below shows the mean absolute error for the PollyVote and each of its component methods, across the remaining days to election. That is, at each day, the PollyGraph shows the error that you would have received by relying on that method’s forecast until Election Day. For example, starting March 15th, which is the day from which forecasts from all six component methods were available, the PollyVote’s average error until Election Day was 2.3 percentage points. This is makes it rank third after citizen forecasts (MAE: 1.1 percentage points) and and econometric models (MAE: 1.8 percentage points). Prediction markets were least accurate and missed on average by 5.9 points. (Note that these numbers will still change as the final vote tally changes. The chart below will always show the latest figures.)

NOTE: THE CHART IS CURRENTLY BROKEN

The relative accuracy of the different methods changes as we get closer to the election. For example, the average of econometric models became more accurate, whereas the accuracy of citizen forecasts decreased somewhat.

Initial thoughts

It’s been a long night and it’s still too early to draw conclusions. But Polly the forecasting parrot feels miserable about her miss. While she did of course perform as well as the typical forecast – which is the minimum what you would expect from a combined forecast – she did not outperform the best individual method as in previous elections. We will work hard to find out what went wrong and what information we missed in order to provide her with even better information next time.

But a look at the method’s relative accuracy in predicting the vote shares provides some interesting first insights. Prediction markets, which were among the most accurate methods historically, experienced a huge error. In contrast, econometric models, which were among the least accurate methods over the last six elections, outperformed all other methods in 2016.

Yesterday, probably no one – us included – would have thought that the econometric models component would turn out to be most accurate. Some forecasters didn’t even trust their own models. Others used the models’ forecasts only as a benchmark to estimate how many votes Trump would cost the Republican party.

These results conform to what we found in prior research on combining forecasts:

  1. The accuracy of different methods changes over time and across elections. There is simply no one best method to predict elections. The methods’ relative accuracy for forecasting an election strongly depends on the context and the idiosyncrasies of that particular election.
  2. It’s very difficult to determine a priori which method will provide the best forecasts. This is also why it makes little sense to weight forecasts based on their historical accuracy, particularly in applications such as election forecasting, where we have limited historical data to derive valid estimates about the method’s relative performance.

This is why Polly’s method of combining different forecasts is still valuable, and perhaps even more important in the future. First, it’s generally not a good idea to trust any single forecast. Second, in aggregating all the different forecasts that are out there, the PollyVote enables us to learn about the accuracy of the various methods – and the biases they may entail – over time.

If we can better understand the conditions under which different methods work best, we might be able to use this information to improve the accuracy of our combined forecast and to better estimate its surrounding uncertainty. In fact, estimating uncertainty is one of the most interesting areas for future research when it comes to combining forecasts.

We will provide additional analyses and thoughts over the next days and weeks. So stay tuned.

4 thoughts on “A first post-mortem”

  1. The pollsters probably didn’t adequately compensate for the pro-democratic bias of their polling.

    Polling and surveys tend to be answered by younger, more democrat leaning people.

    Perhaps pollsters shouldn’t have phoned up and asked to speak to the youngest member of the household.

    The liberal press also liked to share the same story, which may have led to some double counting.

    Much kudos for keeping the Clinton prediction up, even though it is embarrassing for you. This is where science is done, when you find your predictions to be wrong.

  2. The point of prediction markets is that they optimally aggregate information. If you believe that, you shouldn’t be aggregating them with anything else. If you don’t believe it, you probably shouldn’t aggregate them with anything else, either.

    Everybody relied too much on polls. I confess I don’t understand pollology. I looked at them, noted the share of undecided/other was always and everywhere larger than the margin between the candidates, and decided I’d better ignore them. Apparently that’s usually the case, even when polls work, so I guess I don’t understand something.

    What is clear is that *all* polls systematically failed to reach Trump voters. Trump voters misrepresenting themselves in polls was not decisive; failure to poll them at all was.

    That’s because, to be added to phone lists or even download an app or click on a display ad, you’ve got to be politically active. Trump voters didn’t lie, they laid low.

    When, for fear of persecution, shy voters of this kind are present, polls will *never* reach them.

    But fundamental models like those of Norpoth and Abramowitz should perform better than polls in all cases. So just use those — perhaps modified to predict the electoral college vote instead of the (meaningless) popular vote.

Leave a Reply

Your email address will not be published. Required fields are marked *

*