First analysis of the PollyVote’s 2016 popular vote forecast

Since its first launch last January, the combined PollyVote forecast has consistently – and correctly – predicted Hillary Clinton to win the popular vote. And so she did, albeit barely.

In this preliminary post-mortem of how the PollyVote performed in predicting the 2016 popular vote, we compare how each component did relative to the other and to its record on the last six elections. The analysis is based on current projections according to which Clinton will end up winning 50.9% of the popular two-party vote.

Polls

Across the last 100 days, which is the time frame we usually look at in our publications, the MAE of combined vote-intention polls was 1.8 percentage points. That is, polls were in fact considerably more accurate than in previous elections.

polly_pm1

The problem is that there were large polling errors in certain key states, such as Michigan, Wisconsin, and Pennsylvania, that ended up deciding the election in the Electoral College and which no forecaster got right. We will do a separate analysis on whether the PollyVote reduced the error in the EC forecast in due time.

Prediction markets

In prediction markets traders bet on the outcome of an election, and the betting quotes provide a forecast of what is going to happen. Depending on the accuracy of their individual predictions, participants can either win or lose money, and thus have an incentive to be right. They should only participate if they think they have information that improves the current market forecast.

As in previous years, the PollyVote relied on the Iowa Electronic Markets’ (IEM) for predicting the popular vote. The IEM is operated by the Business School of the University of Iowa for teaching and research purposes. The market is relatively low volume and participants are not allowed to invest more than $500. The IEM is the only prediction market for forecasting the vote share, so one cannot combine across markets. We did, however, combine across time by calculating one-week averages to protect against short-term spikes.

Across the last six elections from 1992 to 2012, the IEM vote-share market was the second most accurate among the PollyVote’s components, after citizen forecasts. In 2016, however, the IEM performed poorly and provided the worst forecasts of all components throughout the campaign by wildly overestimated Clinton’s vote share.

Prior research shows that participants in these markets tend to be well educated and belong to middle and upper income groups. It may have been the case that this certain demographic group allowed their wishes to get the better of their judgment.

Expert judgment

Asking experts to predict what is going to happen is one of the oldest forecasting methods available. When it comes to forecasting elections, for example, experts have knowledge and expertise about how to read and interpret polls. In particular, experts know that polls have errors. Thus, they could be expected to be able to figure out in which direction the error is, and thus revise the forecast in the right direction.

In 2016, the combined expert forecast overestimated Clinton’s popular vote share relative to polls in five of the six surveys conducted since the parties’ conventions. On average, the expert forecast was 0.6 percentage points higher than the combined polls. In other words, the expert consensus was that the polls underestimestated Clinton’s support.

Citizen forecasts

As in previous elections, a forecast derived from survey respondents’ answer to a simple question “Who will win?” was among the most accurate methods for predicting the 2016 popular vote. Across the final 100 days before the election, citizen forecasts missed on average by only 1.4 percentage points, an error that is only slightly higher than the average across the six previous elections.

Models

The PollyVote used combined forecasts from two separate groups of models: index and econometric models.

Index models are based on the idea of prospective voting. That is, they rely on the idea that voters evaluate the candidates and how they stand on the issues when deciding upon for whom to vote. On average, the five available index models overestimated Clinton’s support, particularly due to two models that were especially far off (the bio-index and the issue-index). Three of these models (e.g., the big-issue model, the Issues and Leaders model, and the Keys to the White House) were close the final election outcome.

In contrast, most political-economic models are based on the idea of retrospective voting. That is, voters are expected to look back at how well the incumbent government has done its job, particularly in handling the economy. In addition, many models include some measure about how long the incumbent party has been in office to account for Americans’ desire for change, and some also include a measure of the president’s popularity. These  models predicted a very close race, at least on average, and thus were the most accurate component method this election, which is the first time that has happened since 1992. That said, the final forecasts from the 18 individual models differed by as much as 10-points, ranging from 44.0% to 53.9% of the two-party vote.

Combining forecasts

We know from prior research that the relative accuracy of the different methods varies from one election to the next, not just for election forecasting but also in other fields. We can see this again in 2016. Prediction markets, which were among the most accurate methods historically, were dramatically off, while econometric models, historically high in error, turned out to be more accurate this time. That is one of the reasons why combining forecasts usually works well. It’s extremely difficult to predict ex ante which method will end up being most accurate.

In 2016, five of the six component methods erred in the same direction, over-predicting Clinton’s vote share by an average of more than 2 percentage points. As already mentioned, the IEM, in particular, was way of the mark. The only component method that slightly underestimated Clinton’s support were econometric models. As a result, there was little “bracketing” of the true value, which limits the benefits of combining forecasts.

Given the little bracketing, the PollyVote performed only slightly better than the typical forecast. It performed worse than econometric models, citizen forecasts, and polls but outperformed expert judgment, index models, and prediction markets.

The principle of combining forecasts does not claim that the combined forecast will always be more accurate than any of its components. While that can happen – and did happen, for instance, in 2004 and 2012 – it is unlikely to happen in any single election.

The claim is that over time, as the relative accuracy of other methods varies, the combined forecast will outperform its components. This can be seen by looking at the mean absolute forecast errors across all seven elections, including this year’s. On average, the PollyVote error was lower than any of its components.

polly_pm2

Interestingly, citizen forecasts performed nearly as well as the PollyVote. So why not just use this one method in the future, you might ask? One major advantage of combining forecasts is that it’s often among the most accurate methods and, most importantly, it avoids large errors. There is not guarantee that citizen forecasts will perform as well in future elections.

How to improve

At the PollyVote, we are currently reviewing what we can learn from this election. This process includes reviewing which forecasts to include, how to better combine them, and how to better communicate their surrounding uncertainty.

The PollyVote aims at communicating scientific principles that can be used by any organization.

  1. We do know from prior research that the combined forecast will always be at least as accurate than the typical component forecasts, in any single election. So, it prevents you from making large errors.
  2. We also know that, over time and across many elections, the combined forecast will be among the most accurate forecasts available, because the performance of individual forecasts varies wildly.
  3. If you accept that it is extremely difficult to predict which forecast will turn out to be most accurate in a particular election, there is no better way to forecasting than combining forecasts.

The PollyVote forecasters are (in alphabetical order): J. Scott Armstrong, Alfred G. Cuzán, Andreas Graefe, and Randall J. Jones Jr.

A terrible day for election forecasters. Where are the winners?

The 2016 presidential election result may well be one of the biggest upsets in the history of election forecasting. Usually, after an election, people look for the best forecasters. This time, it’s hard to find anyone who got it right, us included.

Regarding the Electoral College, PollyVote combined state-level forecasts from 20 different sources, none of which predicted Trump to win a majority of electoral votes.

Now, people are pointing to the fact that some models did in fact predict Trump to win, such as those by Alan Abramowitz or Helmut Norpoth.

Yet, these models predicted Trump to win the popular vote, which, according to the latest projections, he most likely won’t. Norpoth’s model, for example, predicted Trump to gain 47.5% of the two-party vote and thus might miss by about three points, which is a larger error than most of the other econometric models will end up with.

So are there any winners? Well, you could look at the individual models’ vote share predictions to judge their accuracy. For example, Jim Campbell’s well-known trial-heat model predicted Clinton to gain 50.7% of the vote and might thus be very close. But then, these models don’t provide us with the most important information of who will become president. So, no winners in sight.

A first post-mortem

Based on the latest results, Donald Trump won the election with 305 electoral votes (vs. 233 for Hillary Clinton). Most likely, the final outcome will show a split between the Electoral College and the national popular vote. The New York Times currently projects that Clinton will win the popular vote by 1.3 percentage points.

These results are a major upset for the PollyVote and other election forecasters, as virtually no one saw this coming.

Electoral College

The Electoral College outcome came at a big surprise. The PollyVote’s final Electoral College forecast predicted Clinton to win 323 electoral votes, compared to 215 for Trump, which was very much in line with other forecasters.

For these state-level forecasts, the PollyVote combined predictions from eight poll aggregators, five models, four sources of expert judgment, two prediction markets, and one source of citizen forecasts.

Not a single one of these twenty different sources predicted that Trump would win a majority of the electoral votes! Accordingly, the combined forecast failed as well.

Popular Vote

The final PollyVote forecast predicted Clinton to win the popular vote by 5 percentage points, 52.5% vs. 47.5%. Based on the latest projections, Clinton is expected to win the popular vote by about 1.2 points, which would translate to about 50.6% of the two-party vote. If these numbers are correct, the PollyVote will have missed by 1.9 percentage points, which is more than three times the average error from previous elections.

However, the accuracy of forecasts shouldn’t just be judged based on final predictions. Rather, one should look at which forecasting method has performed best over the course of the campaign.

The PollyGraph below shows the mean absolute error for the PollyVote and each of its component methods, across the remaining days to election. That is, at each day, the PollyGraph shows the error that you would have received by relying on that method’s forecast until Election Day. For example, starting March 15th, which is the day from which forecasts from all six component methods were available, the PollyVote’s average error until Election Day was 2.3 percentage points. This is makes it rank third after citizen forecasts (MAE: 1.1 percentage points) and and econometric models (MAE: 1.8 percentage points). Prediction markets were least accurate and missed on average by 5.9 points. (Note that these numbers will still change as the final vote tally changes. The chart below will always show the latest figures.)

NOTE: THE CHART IS CURRENTLY BROKEN

The relative accuracy of the different methods changes as we get closer to the election. For example, the average of econometric models became more accurate, whereas the accuracy of citizen forecasts decreased somewhat.

Initial thoughts

It’s been a long night and it’s still too early to draw conclusions. But Polly the forecasting parrot feels miserable about her miss. While she did of course perform as well as the typical forecast – which is the minimum what you would expect from a combined forecast – she did not outperform the best individual method as in previous elections. We will work hard to find out what went wrong and what information we missed in order to provide her with even better information next time.

But a look at the method’s relative accuracy in predicting the vote shares provides some interesting first insights. Prediction markets, which were among the most accurate methods historically, experienced a huge error. In contrast, econometric models, which were among the least accurate methods over the last six elections, outperformed all other methods in 2016.

Yesterday, probably no one – us included – would have thought that the econometric models component would turn out to be most accurate. Some forecasters didn’t even trust their own models. Others used the models’ forecasts only as a benchmark to estimate how many votes Trump would cost the Republican party.

These results conform to what we found in prior research on combining forecasts:

  1. The accuracy of different methods changes over time and across elections. There is simply no one best method to predict elections. The methods’ relative accuracy for forecasting an election strongly depends on the context and the idiosyncrasies of that particular election.
  2. It’s very difficult to determine a priori which method will provide the best forecasts. This is also why it makes little sense to weight forecasts based on their historical accuracy, particularly in applications such as election forecasting, where we have limited historical data to derive valid estimates about the method’s relative performance.

This is why Polly’s method of combining different forecasts is still valuable, and perhaps even more important in the future. First, it’s generally not a good idea to trust any single forecast. Second, in aggregating all the different forecasts that are out there, the PollyVote enables us to learn about the accuracy of the various methods – and the biases they may entail – over time.

If we can better understand the conditions under which different methods work best, we might be able to use this information to improve the accuracy of our combined forecast and to better estimate its surrounding uncertainty. In fact, estimating uncertainty is one of the most interesting areas for future research when it comes to combining forecasts.

We will provide additional analyses and thoughts over the next days and weeks. So stay tuned.

Final PollyVote forecast: Clinton will win

All numbers are in and, according to the PollyVote, Hillary Clinton will become the first female president of the United States.

Popular and Electoral Vote Prediction

Clinton will win the popular vote by 5.0 percentage points in the two-party vote (52.5% vs. 47.5%). Based on the PollyVote’s historical error, Clinton’s chance to win the popular vote is above 99%. In terms of the Electoral College, Polly predicts Clinton to receive 323 electoral votes compared to 215 for Trump.

Forecast over the course of the campaign

Forecasting is useful only when it can aid decision-making. Thus, Election Eve forecasts are only for entertainment and betting— and writing victory and concession speeches.

The PollyVote has predicted Clinton to win beginning with its first release in early January 2016. Since the first Super Tuesday on March 1st, when it became clear that Trump would become the Republican nominee, Clinton’s lead in the PollyVote forecast averaged 5.6 points. Throughout the course of the campaign, it ranged from a minimum of 3.8 points to a maximum of 8.2 points. In other words, the PollyVote never had Trump close to winning.

Since its launch in 2004, the PollyVote has always correctly predicted the winner on any given day, months in advance, and with little forecast error. Tonight, we will find out whether Polly the Parrot will keep her clean slate. Check back in the days to come as we will provide analyses of the different forecasting methods’ accuracy.

Political scientists predict Clinton will win 334 electoral votes, compared to 204 for Donald Trump

The PollyVote team has conducted the fourth and final round of its state-level expert survey. According to the forecasts of 638 political scientists, Hillary Clinton will win 334 Electoral Votes, compared to 204 for Donald Trump.

This Electoral College prediction thus differs from the results of the previous survey round,  when the experts’ aggregate forecast was that Clinton would gain 358 Electoral Votes, compared to 180 for Trump. The difference is due to Iowa and Ohio, which the experts now see leaning towards the Republicans. For Arizona, the experts predict essentially a pure tossup in both median winning probabilities and vote shares. However, since the average forecast for Clinton’s vote share is slightly above 50%, Arizona is called for the Democrats.

The following map visualizes the experts’ median estimates regarding Clinton’s chance of winning each state.

Nebraska and Maine deserve special attention, since these two states allocate two Electoral Votes to the popular vote winner plus one each to the popular vote winner in each Congressional district. The experts predict that all districts will go with the state-wide popular with two exceptions.

  • In Nebraska’s 2nd district, the Democratic candidate is expected to win 51.0% of the two-party vote.
  • In Maine’s 2nd district, the Republican candidate is expected to win the popular vote with a two-party vote share of 50.6%.

Method

We reached out to political scientists across the country and asked them two short questions:

  1. What share of the vote do you expect the nominees to receive in your home state?
  2. What do you think is Hillary Clinton’s chance of winning the election in your home state?

For experts who regard Nebraska or Maine as their home state, we additionally asked them to predict the outcome in each congressional district.

The survey was conducted from November 6 to 7. A total of 638 experts made estimates as requested. The number of experts by state ranged from 2 to 44. The table below shows the number of respondents per state as well as the median answer for each question.

    Clinton’s predicted
State N Chance of winning Two-party vote
District of Columbia 16 100% 90.4
Hawaii 2 100% 72.3
Vermont 11 99% 67.0
Maryland 22 99% 63.2
Massachusetts 17 99% 66.7
New York 23 99% 63.2
California 21 99% 62.4
Illinois 23 99% 57.9
Connecticut 19 99% 57.9
New Jersey 4 99% 58.5
Washington 7 98% 56.5
Oregon 8 98% 56.8
Delaware 13 98% 62.8
Rhode Island 6 97% 58.8
New Mexico 4 97% 53.0
Minnesota 13 95% 53.2
Maine 8 90% 53.7
Wisconsin 14 89% 52.7
Virginia 44 85% 52.7
Nevada 8 85% 52.4
Michigan 12 83% 51.6
Pennsylvania 19 80% 52.1
Colorado 10 71% 52.0
New Hampshire 10 68% 52.7
Florida 19 55% 50.6
North Carolina 23 55% 51.1
Arizona 13 50% 50.0
Ohio 15 48% 49.5
Iowa 18 43% 48.9
Alaska 4 30% 47.0
Georgia 24 29% 47.6
Missouri 12 25% 47.5
Utah 15 15% 44.4
South Carolina 12 15% 46.3
Kansas 9 15% 43.3
Alabama 6 13% 37.9
Indiana 16 10% 44.2
Texas 22 10% 45.7
Kentucky 5 8% 41.2
Montana 5 5% 41.6
South Dakota 3 5% 42.1
Louisiana 8 5% 43.2
North Dakota 6 4% 36.6
Arkansas 8 3% 41.1
Mississippi 12 2% 42.0
Idaho 15 1% 39.0
Tennessee 5 1% 42.1
West Virginia 9 0% 35.5
Nebraska 5 0% 42.1
Wyoming 4 0% 28.8
Oklahoma 11 0% 30.9

Final expert survey: Clinton will win by 4.4 points

The PollyVote team has completed its 13th and final survey of elections experts to forecast the 2016 presidential election. In this survey, conducted between November 6 and 7, 12 academics from a variety of colleges and universities responded.

As in previous rounds, all respondents expect a Clinton win. However, her lead has narrowed again compared to the results of the previous survey conducted a week ago.

Whereas, in late-October, the experts expected that Clinton will win the popular vote by more than 5.5 points, the new average forecast is half a percentage point lower, at 52.2% of the two-party vote (or about a 4.4-point margin). The individual forecasts ranged from 51.0% to 53.3%, with a standard deviation of only 0.8 points.

Polly thanks the experts who participated in this round, namely

  1. Randall Adkins (University of Nebraska Omaha)
  2. Lonna Rae Atkeson (University of New Mexico)
  3. John Geer (Vanderbilt University)
  4. Sandy Maisel (Colby College)
  5. Michael Martinez (University of Florida)
  6. Thomas Patterson (Harvard University)
  7. Gerald Pomper (Rutgers University)
  8. David Redlawsk (University of Delaware)
  9. Larry Sabato (University of Virginia)
  10. Michael Tesler (University of California, Irvine)
  11. Charles Walcott (Virginia Tech)

and one expert who preferred to remain anonymous.

New Economist poll: Clinton holds slim lead

Results of a new national poll administered by Economist were released. The poll asked respondents for whom they will vote: Republican Donald Trump or Democrat Hillary Clinton.

Economist poll results
49

Clinton

45

Trump

Of those who responded, 49.0% said that they plan to vote for former Secretary of State Hillary Clinton, whereas 45.0% said that they would give their vote to real estate developer Donald Trump.

The poll was conducted between November 4 and November 7. The sample size was 3669 participants. There is a sampling error of +/-1.7 percentage points. Considering this error margin, the gap between both candidates is statistically significant.

Putting the results in context

Individual polls should be treated with caution, since they can include substantial errors. Instead of relying on results from single polls, research in forecasting recommends to use combined polls or, even better, the combined PollyVote forecast that relies on different methods and data.

In order to make the results comparable to benchmark forecasts, one can translate them into shares of the two-party vote. This procedure results in values of 52.1% for Clinton and 47.9% for Trump. On November 1 Clinton obtained only 51.6% in the Economist poll and Trump obtained 48.4%.

Comparison to other polls

An average of recent polls sees Clinton at 52.0% of the two-party vote. This value is 0.1 percentage points lower than her respective numbers in the Economist poll. This deviation is within the poll's error margin, which suggests that the poll is not an outlier.

Comparison to the combined PollyVote

The latest PollyVote expects Clinton to gain 52.6% of the two-party vote. This means that Polly's forecast is 0.5 points above her polling numbers. Again, a look at the poll's margin of error indicates that this deviation is negligible.

This article was automatically generated by the PollyBot, which uses algorithms developed by AX Semantics to generate text from data stored in our API. The exact dataset underlying this particular article can be found here.

Please let us know if you find any typos, missing words, or grammatical errors. Your feedback helps us to further improve the quality of the texts.

New York: Overwhelming lead for Clinton in latest Siena poll

Siena published the results of a new poll. In this poll, respondents from New York were asked for whom they will vote: Donald Trump or Hillary Clinton.

Siena poll results
51

Clinton

30

Trump

The results show that 51.0% of participants will give their vote to former New York Senator Hillary Clinton, while 30.0% plan to vote for billionaire Donald Trump.

The poll was conducted from September 11 to September 15. A total of 600 likely voters responded. There is a sampling error of +/-5.0 percentage points. Considering this error margin, the gap between both candidates is statistically significant.

Putting the results in context

Single polls can incorporate large errors, and should be treated with caution. Rather, one should check how a poll's results compare to benchmark forecasts.

For the following analysis, we translate Clinton's and Trump's raw poll numbers into shares of the two-party vote. This yields figures of 63.0% for Clinton and 37.0% for Trump. The most recent PollyVote foresees Clinton to gain 63.0% of the two-party vote in New York.

This article was automatically generated by the PollyBot, which uses algorithms developed by AX Semantics to generate text from data stored in our API. The exact dataset underlying this particular article can be found here.

Please let us know if you find any typos, missing words, or grammatical errors. Your feedback helps us to further improve the quality of the texts.

Latest KSTP/SurveyUSA poll in Minnesota: Clinton and Trump in a dead heat

KSTP/SurveyUSA released the results of a new poll, in which respondents from Minnesota were asked for whom they will vote: Donald Trump or Hillary Clinton.

KSTP/SurveyUSA poll results
42

Clinton

45

Trump

The results show that 42.0% of participants said that they would cast a ballot for former First Lady Hillary Clinton, while 45.0% intend to vote for businessman Donald Trump.

The poll was conducted from October 29 to November 2. A total of 516 registered voters responded. The margin of error is +/-4.4 points. This means that the poll results for both parties' candidates do not differ significantly.

Putting the results in context

Individual polls often incorporate large errors, which is why they should be interpreted with caution. Rather than trusting the results from single polls, one should rely on combined polls or, even better, the combined PollyVote forecast that draws upon forecasts from different methods, each of which draws upon different data.

For the following comparison, we convert the candidates' raw poll numbers into two-party vote shares. This yields figures of 48.3% for Clinton and 51.7% for Trump. For comparison: 55.8% was obtained by Clinton in the KSTP/SurveyUSA poll on October 25, for Trump this number was only 44.2%.

Results in comparison to other polls

Looking at an average of Minnesota polls, Trump's two-party vote share is currently at 46.4%. Compared to his numbers in the KSTP/SurveyUSA poll Trump's poll average is 5.3 percentage points worse. This margin is outside the poll's margin of error, which suggests that the poll is an outlier.

Comparison to the combined PollyVote

The current PollyVote forecasts Trump to gain 44.5% of the two-party vote in Minnesota. That is, the PollyVote is 7.2 points below his polling numbers. Again, a look at the poll's sampling error reveals that this difference is significant.

This article was automatically generated by the PollyBot, which uses algorithms developed by AX Semantics to generate text from data stored in our API. The exact dataset underlying this particular article can be found here.

Please let us know if you find any typos, missing words, or grammatical errors. Your feedback helps us to further improve the quality of the texts.

Star Tribune/Mason-Dixon poll in Minnesota: Clinton with 13 points lead

Star Tribune/Mason-Dixon published the results of a new poll. In this poll, respondents from Minnesota were asked for whom they will vote: Democrat candidate Hillary Clinton or Republican candidate Donald Trump.

Star Tribune/Mason-Dixon poll results
48

Clinton

35

Trump

Of those who answered the question, 48.0% said that they would vote for former New York Senator Hillary Clinton, while 35.0% revealed that they would give their vote to billionaire Donald Trump.

The poll was carried out from April 25 to April 27 with 800 registered voters. There is a sampling error of +/-3.5 percentage points. Considering this error margin, the gap between both candidates is statistically significant.

Putting the results in context

As a general rule, however, you should not have too much faith in the results of single polls, as they may contain large errors. Instead of trusting the results from single polls, forecasting research recommends to look at combined polls or, even better, a combined forecast that draws upon different methods and data.

In order to make the results comparable to forecasts from benchmark methods, you can translate them into two-party vote shares. The resulting figures are 57.8% for Clinton and 42.2% for Trump. On October 22 Clinton obtained only 54.7% in the Star Tribune/Mason-Dixon poll and Trump obtained 45.4%.

Comparison to other polls

An average of recent polls in Minnesota has Clinton at 53.6% of the two-party vote. This value is 4.2 percentage points lower than her corresponding numbers in the Star Tribune/Mason-Dixon poll. This difference is outside the poll's sampling error, which suggests that the poll is an outlier.

Comparison to the combined PollyVote

The most recent PollyVote foresees Clinton to gain 55.5% of the two-party vote in Minnesota. That is, the PollyVote forecast is 2.3 points below her polling numbers. The PollyVote forecast is therefore in line with the poll's error margin.

This article was automatically generated by the PollyBot, which uses algorithms developed by AX Semantics to generate text from data stored in our API. The exact dataset underlying this particular article can be found here.

Please let us know if you find any typos, missing words, or grammatical errors. Your feedback helps us to further improve the quality of the texts.