First analysis of the PollyVote’s 2016 popular vote forecast

Since its first launch last January, the combined PollyVote forecast has consistently – and correctly – predicted Hillary Clinton to win the popular vote. And so she did, albeit barely.

In this preliminary post-mortem of how the PollyVote performed in predicting the 2016 popular vote, we compare how each component did relative to the other and to its record on the last six elections. The analysis is based on current projections according to which Clinton will end up winning 50.9% of the popular two-party vote.

Polls

Across the last 100 days, which is the time frame we usually look at in our publications, the MAE of combined vote-intention polls was 1.8 percentage points. That is, polls were in fact considerably more accurate than in previous elections.

polly_pm1

The problem is that there were large polling errors in certain key states, such as Michigan, Wisconsin, and Pennsylvania, that ended up deciding the election in the Electoral College and which no forecaster got right. We will do a separate analysis on whether the PollyVote reduced the error in the EC forecast in due time.

Prediction markets

In prediction markets traders bet on the outcome of an election, and the betting quotes provide a forecast of what is going to happen. Depending on the accuracy of their individual predictions, participants can either win or lose money, and thus have an incentive to be right. They should only participate if they think they have information that improves the current market forecast.

As in previous years, the PollyVote relied on the Iowa Electronic Markets’ (IEM) for predicting the popular vote. The IEM is operated by the Business School of the University of Iowa for teaching and research purposes. The market is relatively low volume and participants are not allowed to invest more than $500. The IEM is the only prediction market for forecasting the vote share, so one cannot combine across markets. We did, however, combine across time by calculating one-week averages to protect against short-term spikes.

Across the last six elections from 1992 to 2012, the IEM vote-share market was the second most accurate among the PollyVote’s components, after citizen forecasts. In 2016, however, the IEM performed poorly and provided the worst forecasts of all components throughout the campaign by wildly overestimated Clinton’s vote share.

Prior research shows that participants in these markets tend to be well educated and belong to middle and upper income groups. It may have been the case that this certain demographic group allowed their wishes to get the better of their judgment.

Expert judgment

Asking experts to predict what is going to happen is one of the oldest forecasting methods available. When it comes to forecasting elections, for example, experts have knowledge and expertise about how to read and interpret polls. In particular, experts know that polls have errors. Thus, they could be expected to be able to figure out in which direction the error is, and thus revise the forecast in the right direction.

In 2016, the combined expert forecast overestimated Clinton’s popular vote share relative to polls in five of the six surveys conducted since the parties’ conventions. On average, the expert forecast was 0.6 percentage points higher than the combined polls. In other words, the expert consensus was that the polls underestimestated Clinton’s support.

Citizen forecasts

As in previous elections, a forecast derived from survey respondents’ answer to a simple question “Who will win?” was among the most accurate methods for predicting the 2016 popular vote. Across the final 100 days before the election, citizen forecasts missed on average by only 1.4 percentage points, an error that is only slightly higher than the average across the six previous elections.

Models

The PollyVote used combined forecasts from two separate groups of models: index and econometric models.

Index models are based on the idea of prospective voting. That is, they rely on the idea that voters evaluate the candidates and how they stand on the issues when deciding upon for whom to vote. On average, the five available index models overestimated Clinton’s support, particularly due to two models that were especially far off (the bio-index and the issue-index). Three of these models (e.g., the big-issue model, the Issues and Leaders model, and the Keys to the White House) were close the final election outcome.

In contrast, most political-economic models are based on the idea of retrospective voting. That is, voters are expected to look back at how well the incumbent government has done its job, particularly in handling the economy. In addition, many models include some measure about how long the incumbent party has been in office to account for Americans’ desire for change, and some also include a measure of the president’s popularity. These  models predicted a very close race, at least on average, and thus were the most accurate component method this election, which is the first time that has happened since 1992. That said, the final forecasts from the 18 individual models differed by as much as 10-points, ranging from 44.0% to 53.9% of the two-party vote.

Combining forecasts

We know from prior research that the relative accuracy of the different methods varies from one election to the next, not just for election forecasting but also in other fields. We can see this again in 2016. Prediction markets, which were among the most accurate methods historically, were dramatically off, while econometric models, historically high in error, turned out to be more accurate this time. That is one of the reasons why combining forecasts usually works well. It’s extremely difficult to predict ex ante which method will end up being most accurate.

In 2016, five of the six component methods erred in the same direction, over-predicting Clinton’s vote share by an average of more than 2 percentage points. As already mentioned, the IEM, in particular, was way of the mark. The only component method that slightly underestimated Clinton’s support were econometric models. As a result, there was little “bracketing” of the true value, which limits the benefits of combining forecasts.

Given the little bracketing, the PollyVote performed only slightly better than the typical forecast. It performed worse than econometric models, citizen forecasts, and polls but outperformed expert judgment, index models, and prediction markets.

The principle of combining forecasts does not claim that the combined forecast will always be more accurate than any of its components. While that can happen – and did happen, for instance, in 2004 and 2012 – it is unlikely to happen in any single election.

The claim is that over time, as the relative accuracy of other methods varies, the combined forecast will outperform its components. This can be seen by looking at the mean absolute forecast errors across all seven elections, including this year’s. On average, the PollyVote error was lower than any of its components.

polly_pm2

Interestingly, citizen forecasts performed nearly as well as the PollyVote. So why not just use this one method in the future, you might ask? One major advantage of combining forecasts is that it’s often among the most accurate methods and, most importantly, it avoids large errors. There is not guarantee that citizen forecasts will perform as well in future elections.

How to improve

At the PollyVote, we are currently reviewing what we can learn from this election. This process includes reviewing which forecasts to include, how to better combine them, and how to better communicate their surrounding uncertainty.

The PollyVote aims at communicating scientific principles that can be used by any organization.

  1. We do know from prior research that the combined forecast will always be at least as accurate than the typical component forecasts, in any single election. So, it prevents you from making large errors.
  2. We also know that, over time and across many elections, the combined forecast will be among the most accurate forecasts available, because the performance of individual forecasts varies wildly.
  3. If you accept that it is extremely difficult to predict which forecast will turn out to be most accurate in a particular election, there is no better way to forecasting than combining forecasts.

The PollyVote forecasters are (in alphabetical order): J. Scott Armstrong, Alfred G. Cuzán, Andreas Graefe, and Randall J. Jones Jr.

A terrible day for election forecasters. Where are the winners?

The 2016 presidential election result may well be one of the biggest upsets in the history of election forecasting. Usually, after an election, people look for the best forecasters. This time, it’s hard to find anyone who got it right, us included.

Regarding the Electoral College, PollyVote combined state-level forecasts from 20 different sources, none of which predicted Trump to win a majority of electoral votes.

Now, people are pointing to the fact that some models did in fact predict Trump to win, such as those by Alan Abramowitz or Helmut Norpoth.

Yet, these models predicted Trump to win the popular vote, which, according to the latest projections, he most likely won’t. Norpoth’s model, for example, predicted Trump to gain 47.5% of the two-party vote and thus might miss by about three points, which is a larger error than most of the other econometric models will end up with.

So are there any winners? Well, you could look at the individual models’ vote share predictions to judge their accuracy. For example, Jim Campbell’s well-known trial-heat model predicted Clinton to gain 50.7% of the vote and might thus be very close. But then, these models don’t provide us with the most important information of who will become president. So, no winners in sight.

A first post-mortem

Based on the latest results, Donald Trump won the election with 305 electoral votes (vs. 233 for Hillary Clinton). Most likely, the final outcome will show a split between the Electoral College and the national popular vote. The New York Times currently projects that Clinton will win the popular vote by 1.3 percentage points.

These results are a major upset for the PollyVote and other election forecasters, as virtually no one saw this coming.

Electoral College

The Electoral College outcome came at a big surprise. The PollyVote’s final Electoral College forecast predicted Clinton to win 323 electoral votes, compared to 215 for Trump, which was very much in line with other forecasters.

For these state-level forecasts, the PollyVote combined predictions from eight poll aggregators, five models, four sources of expert judgment, two prediction markets, and one source of citizen forecasts.

Not a single one of these twenty different sources predicted that Trump would win a majority of the electoral votes! Accordingly, the combined forecast failed as well.

Popular Vote

The final PollyVote forecast predicted Clinton to win the popular vote by 5 percentage points, 52.5% vs. 47.5%. Based on the latest projections, Clinton is expected to win the popular vote by about 1.2 points, which would translate to about 50.6% of the two-party vote. If these numbers are correct, the PollyVote will have missed by 1.9 percentage points, which is more than three times the average error from previous elections.

However, the accuracy of forecasts shouldn’t just be judged based on final predictions. Rather, one should look at which forecasting method has performed best over the course of the campaign.

The PollyGraph below shows the mean absolute error for the PollyVote and each of its component methods, across the remaining days to election. That is, at each day, the PollyGraph shows the error that you would have received by relying on that method’s forecast until Election Day. For example, starting March 15th, which is the day from which forecasts from all six component methods were available, the PollyVote’s average error until Election Day was 2.3 percentage points. This is makes it rank third after citizen forecasts (MAE: 1.1 percentage points) and and econometric models (MAE: 1.8 percentage points). Prediction markets were least accurate and missed on average by 5.9 points. (Note that these numbers will still change as the final vote tally changes. The chart below will always show the latest figures.)

NOTE: THE CHART IS CURRENTLY BROKEN

The relative accuracy of the different methods changes as we get closer to the election. For example, the average of econometric models became more accurate, whereas the accuracy of citizen forecasts decreased somewhat.

Initial thoughts

It’s been a long night and it’s still too early to draw conclusions. But Polly the forecasting parrot feels miserable about her miss. While she did of course perform as well as the typical forecast – which is the minimum what you would expect from a combined forecast – she did not outperform the best individual method as in previous elections. We will work hard to find out what went wrong and what information we missed in order to provide her with even better information next time.

But a look at the method’s relative accuracy in predicting the vote shares provides some interesting first insights. Prediction markets, which were among the most accurate methods historically, experienced a huge error. In contrast, econometric models, which were among the least accurate methods over the last six elections, outperformed all other methods in 2016.

Yesterday, probably no one – us included – would have thought that the econometric models component would turn out to be most accurate. Some forecasters didn’t even trust their own models. Others used the models’ forecasts only as a benchmark to estimate how many votes Trump would cost the Republican party.

These results conform to what we found in prior research on combining forecasts:

  1. The accuracy of different methods changes over time and across elections. There is simply no one best method to predict elections. The methods’ relative accuracy for forecasting an election strongly depends on the context and the idiosyncrasies of that particular election.
  2. It’s very difficult to determine a priori which method will provide the best forecasts. This is also why it makes little sense to weight forecasts based on their historical accuracy, particularly in applications such as election forecasting, where we have limited historical data to derive valid estimates about the method’s relative performance.

This is why Polly’s method of combining different forecasts is still valuable, and perhaps even more important in the future. First, it’s generally not a good idea to trust any single forecast. Second, in aggregating all the different forecasts that are out there, the PollyVote enables us to learn about the accuracy of the various methods – and the biases they may entail – over time.

If we can better understand the conditions under which different methods work best, we might be able to use this information to improve the accuracy of our combined forecast and to better estimate its surrounding uncertainty. In fact, estimating uncertainty is one of the most interesting areas for future research when it comes to combining forecasts.

We will provide additional analyses and thoughts over the next days and weeks. So stay tuned.

Final PollyVote forecast: Clinton will win

All numbers are in and, according to the PollyVote, Hillary Clinton will become the first female president of the United States.

Popular and Electoral Vote Prediction

Clinton will win the popular vote by 5.0 percentage points in the two-party vote (52.5% vs. 47.5%). Based on the PollyVote’s historical error, Clinton’s chance to win the popular vote is above 99%. In terms of the Electoral College, Polly predicts Clinton to receive 323 electoral votes compared to 215 for Trump.

Forecast over the course of the campaign

Forecasting is useful only when it can aid decision-making. Thus, Election Eve forecasts are only for entertainment and betting— and writing victory and concession speeches.

The PollyVote has predicted Clinton to win beginning with its first release in early January 2016. Since the first Super Tuesday on March 1st, when it became clear that Trump would become the Republican nominee, Clinton’s lead in the PollyVote forecast averaged 5.6 points. Throughout the course of the campaign, it ranged from a minimum of 3.8 points to a maximum of 8.2 points. In other words, the PollyVote never had Trump close to winning.

Since its launch in 2004, the PollyVote has always correctly predicted the winner on any given day, months in advance, and with little forecast error. Tonight, we will find out whether Polly the Parrot will keep her clean slate. Check back in the days to come as we will provide analyses of the different forecasting methods’ accuracy.

Political scientists predict Clinton will win 334 electoral votes, compared to 204 for Donald Trump

The PollyVote team has conducted the fourth and final round of its state-level expert survey. According to the forecasts of 638 political scientists, Hillary Clinton will win 334 Electoral Votes, compared to 204 for Donald Trump.

This Electoral College prediction thus differs from the results of the previous survey round,  when the experts’ aggregate forecast was that Clinton would gain 358 Electoral Votes, compared to 180 for Trump. The difference is due to Iowa and Ohio, which the experts now see leaning towards the Republicans. For Arizona, the experts predict essentially a pure tossup in both median winning probabilities and vote shares. However, since the average forecast for Clinton’s vote share is slightly above 50%, Arizona is called for the Democrats.

The following map visualizes the experts’ median estimates regarding Clinton’s chance of winning each state.

Nebraska and Maine deserve special attention, since these two states allocate two Electoral Votes to the popular vote winner plus one each to the popular vote winner in each Congressional district. The experts predict that all districts will go with the state-wide popular with two exceptions.

  • In Nebraska’s 2nd district, the Democratic candidate is expected to win 51.0% of the two-party vote.
  • In Maine’s 2nd district, the Republican candidate is expected to win the popular vote with a two-party vote share of 50.6%.

Method

We reached out to political scientists across the country and asked them two short questions:

  1. What share of the vote do you expect the nominees to receive in your home state?
  2. What do you think is Hillary Clinton’s chance of winning the election in your home state?

For experts who regard Nebraska or Maine as their home state, we additionally asked them to predict the outcome in each congressional district.

The survey was conducted from November 6 to 7. A total of 638 experts made estimates as requested. The number of experts by state ranged from 2 to 44. The table below shows the number of respondents per state as well as the median answer for each question.

    Clinton’s predicted
State N Chance of winning Two-party vote
District of Columbia 16 100% 90.4
Hawaii 2 100% 72.3
Vermont 11 99% 67.0
Maryland 22 99% 63.2
Massachusetts 17 99% 66.7
New York 23 99% 63.2
California 21 99% 62.4
Illinois 23 99% 57.9
Connecticut 19 99% 57.9
New Jersey 4 99% 58.5
Washington 7 98% 56.5
Oregon 8 98% 56.8
Delaware 13 98% 62.8
Rhode Island 6 97% 58.8
New Mexico 4 97% 53.0
Minnesota 13 95% 53.2
Maine 8 90% 53.7
Wisconsin 14 89% 52.7
Virginia 44 85% 52.7
Nevada 8 85% 52.4
Michigan 12 83% 51.6
Pennsylvania 19 80% 52.1
Colorado 10 71% 52.0
New Hampshire 10 68% 52.7
Florida 19 55% 50.6
North Carolina 23 55% 51.1
Arizona 13 50% 50.0
Ohio 15 48% 49.5
Iowa 18 43% 48.9
Alaska 4 30% 47.0
Georgia 24 29% 47.6
Missouri 12 25% 47.5
Utah 15 15% 44.4
South Carolina 12 15% 46.3
Kansas 9 15% 43.3
Alabama 6 13% 37.9
Indiana 16 10% 44.2
Texas 22 10% 45.7
Kentucky 5 8% 41.2
Montana 5 5% 41.6
South Dakota 3 5% 42.1
Louisiana 8 5% 43.2
North Dakota 6 4% 36.6
Arkansas 8 3% 41.1
Mississippi 12 2% 42.0
Idaho 15 1% 39.0
Tennessee 5 1% 42.1
West Virginia 9 0% 35.5
Nebraska 5 0% 42.1
Wyoming 4 0% 28.8
Oklahoma 11 0% 30.9

Final expert survey: Clinton will win by 4.4 points

The PollyVote team has completed its 13th and final survey of elections experts to forecast the 2016 presidential election. In this survey, conducted between November 6 and 7, 12 academics from a variety of colleges and universities responded.

As in previous rounds, all respondents expect a Clinton win. However, her lead has narrowed again compared to the results of the previous survey conducted a week ago.

Whereas, in late-October, the experts expected that Clinton will win the popular vote by more than 5.5 points, the new average forecast is half a percentage point lower, at 52.2% of the two-party vote (or about a 4.4-point margin). The individual forecasts ranged from 51.0% to 53.3%, with a standard deviation of only 0.8 points.

Polly thanks the experts who participated in this round, namely

  1. Randall Adkins (University of Nebraska Omaha)
  2. Lonna Rae Atkeson (University of New Mexico)
  3. John Geer (Vanderbilt University)
  4. Sandy Maisel (Colby College)
  5. Michael Martinez (University of Florida)
  6. Thomas Patterson (Harvard University)
  7. Gerald Pomper (Rutgers University)
  8. David Redlawsk (University of Delaware)
  9. Larry Sabato (University of Virginia)
  10. Michael Tesler (University of California, Irvine)
  11. Charles Walcott (Virginia Tech)

and one expert who preferred to remain anonymous.

Experts agree: Clinton will win

The PollyVote team has completed its 12th survey of elections experts to forecast the 2016 presidential election. In this survey, conducted between October 29 and 31, 13 academics from a variety of colleges and universities responded.

As in previous rounds, all respondents expect a Clinton win. However, her lead has narrowed substantially compared to the results of the previous survey conducted two weeks ago.

Whereas, in mid-October, the experts expected that Clinton will win the popular vote by more than nine points, the new average forecast is two percentage points lower, at 52.7% of the two-party vote (or about a 5.5-point margin). The individual forecasts ranged from 51.6% to 54.8%, with a standard deviation of only 0.8 points.

Polly thanks the experts who participated in this round, namely

  1. Randall Adkins (University of Nebraska Omaha)
  2. Lonna Rae Atkeson (University of New Mexico)
  3. George Edwards (Texas A&M University)
  4. Keith Gaddie (University of Oklahoma)
  5. John Geer (Vanderbilt University)
  6. Sandy Maisel (Colby College)
  7. Michael Martinez (University of Florida)
  8. Thomas Patterson (Harvard University)
  9. Gerald Pomper (Rutgers University)
  10. David Redlawsk (University of Delaware)
  11. Larry Sabato (University of Virginia)
  12. Michael Tesler (University of California, Irvine)
  13. Charles Walcott (Virginia Tech)

State-wide survey of political scientists: Clinton now at 358 electoral votes

The PollyVote team has conducted the third round of its state-level expert survey. According to the forecasts of 673 political scientists, Hillary Clinton will win 358 Electoral Votes, compared to 180 for Donald Trump.

This Electoral College prediction thus differs from the results of the first and second survey rounds,  when the experts’ aggregate forecast was that Clinton would gain 347 Electoral Votes, compared to 191 for Trump. The difference is due to Arizona (with its 11 Electoral Votes), which the experts now see leaning slightly towards the Democrats.

The following map visualizes the experts’ median estimates regarding Clinton’s chance of winning each state.

Nebraska and Maine deserve special attention, since these two states allocate two Electoral Votes to the popular vote winner plus one each to the popular vote winner in each Congressional district. The experts predict that all districts will go with the state-wide popular with two exceptions.

  • In Nebraska’s 2nd district, the Democratic candidate is expected to win 50.8% of the two-party vote.
  • In Maine’s 2nd district, the Republican candidate is expected to win the popular vote with a two-party vote share of 51.0%.

Method

We reached out to political scientists across the country and asked them two short questions:

  1. What share of the vote do you expect the nominees to receive in your home state?
  2. What do you think is Hillary Clinton’s chance of winning the election in your home state?

For experts who regard Nebraska or Maine as their home state, we additionally asked them to predict the outcome in each congressional district.

The survey was conducted from October 15 to 19. A total of 673 experts made estimates as requested. The number of experts by state ranged from 2 to 39. The table below shows the number of respondents per state as well as the median answer for each question.

Clinton’s predicted
State N Chance of winning Two-party vote
District of Columbia 15 100% 93.8
Vermont 9 100% 67.4
Hawaii 2 100% 76.5
Maryland 26 100% 67.0
New Mexico 5 100% 59.0
Massachusetts 14 100% 67.6
New York 26 99% 65.2
California 23 99% 63.8
Rhode Island 5 99% 61.1
Washington 12 99% 60.3
Illinois 24 99% 59.7
Connecticut 21 99% 62.4
Oregon 9 95% 57.8
New Jersey 5 95% 63.2
Minnesota 18 95% 55.3
Michigan 12 91% 54.3
Delaware 11 90% 63.2
Wisconsin 14 90% 53.9
Virginia 39 90% 56.0
Pennsylvania 20 90% 54.3
Colorado 8 88% 54.9
Maine 14 84% 54.4
New Hampshire 14 80% 54.2
Nevada 5 75% 53.8
Florida 22 73% 51.6
North Carolina 24 65% 51.3
Ohio 12 58% 51.4
Iowa 19 52% 50.5
Arizona 16 50% 50.5
Missouri 11 45% 47.8
Utah 13 40% 47.6
Georgia 26 39% 48.4
Alaska 5 38% 45.8
South Carolina 14 25% 47.0
Indiana 15 20% 46.4
Texas 19 20% 46.7
Kansas 11 15% 46.0
Montana 5 15% 46.2
Kentucky 4 14% 41.5
Arkansas 12 11% 43.2
South Dakota 6 9% 40.3
North Dakota 6 7% 42.0
West Virginia 9 7% 37.6
Mississippi 14 7% 44.7
Alabama 5 5% 38.9
Louisiana 11 5% 40.9
Idaho 14 5% 37.8
Tennessee 8 4% 43.3
Nebraska 4 3% 42.2
Wyoming 4 1% 30.0
Oklahoma 13 0% 37.5

Clinton at new record high in PollyVote.com expert survey

The PollyVote team has completed its 11th survey of elections experts to forecast the national vote in the 2016 presidential election. In this survey, conducted between October 13 and 15, and thus after the second debate, 14 academics from a variety of colleges and universities responded.

As in the previous round, all respondents expect a Clinton win. However, her lead has widened substantially, to a new record high.

Whereas, in late September, the experts expected that Clinton will win the popular vote by 4 points, the new average forecast is 2.5 percentage points higher, at 54.7% of the two-party vote (or about a 9-point margin). The individual forecasts ranged from 53.3% to 57.4%, with a standard deviation of 1.4 points.

Polly thanks the experts who participated in this round, namely

  1. Randall Adkins (University of Nebraska Omaha)
  2. Lonna Rae Atkeson (University of New Mexico)
  3. John Coleman (University of Minnesota)
  4. George Edwards (Texas A&M University)
  5. Keith Gaddie (University of Oklahoma)
  6. John Geer (Vanderbilt University)
  7. Sandy Maisel (Colby College)
  8. Michael Martinez (University of Florida)
  9. Thomas Patterson (Harvard University)
  10. Gerald Pomper (Rutgers University)
  11. David Redlawsk (University of Delaware)
  12. Larry Sabato (University of Virginia)
  13. Michael Tesler (University of California, Irvine)
  14. Charles Walcott (Virginia Tech)

Experts predict Clinton will win popular vote by 4 points

The PollyVote team has completed its 10th survey of elections experts to forecast the 2016 presidential election. In this survey, conducted between September 28 and 30, and thus after the first presidential debate, 13 academics from a variety of colleges and universities responded.

Similar to the previous round, all respondents expect a Clinton win. However, her lead has narrowed substantially.

Whereas, in late August, the experts expected that Clinton will win the popular vote by 7 points, the new average forecast is 1.3 percentage points lower, at 52.2% of the two-party vote (or about a 4-point margin). The individual forecasts ranged from 51.6% to 53.7%, with a standard deviation of only 0.7 points.

 

Polly thanks the experts who participated in this round, namely

  1. Randall Adkins (University of Nebraska Omaha)
  2. Lonna Rae Atkeson (University of New Mexico)
  3. Keith Gaddie (University of Oklahoma)
  4. John Geer (Vanderbilt University)
  5. Sandy Maisel (Colby College)
  6. Michael Martinez (University of Florida)
  7. Thomas Patterson (Harvard University)
  8. Gerald Pomper (Rutgers University)
  9. David Redlawsk (University of Delaware)
  10. Larry Sabato (University of Virginia)
  11. Michael Tesler (University of California, Irvine)
  12. Charles Walcott (Virginia Tech)

and one expert who preferred to remain anonymous.