First analysis of the PollyVote’s 2016 popular vote forecast

Since its first launch last January, the combined PollyVote forecast has consistently – and correctly – predicted Hillary Clinton to win the popular vote. And so she did, albeit barely.

In this preliminary post-mortem of how the PollyVote performed in predicting the 2016 popular vote, we compare how each component did relative to the other and to its record on the last six elections. The analysis is based on current projections according to which Clinton will end up winning 50.9% of the popular two-party vote.

Polls

Across the last 100 days, which is the time frame we usually look at in our publications, the MAE of combined vote-intention polls was 1.8 percentage points. That is, polls were in fact considerably more accurate than in previous elections.

polly_pm1

The problem is that there were large polling errors in certain key states, such as Michigan, Wisconsin, and Pennsylvania, that ended up deciding the election in the Electoral College and which no forecaster got right. We will do a separate analysis on whether the PollyVote reduced the error in the EC forecast in due time.

Prediction markets

In prediction markets traders bet on the outcome of an election, and the betting quotes provide a forecast of what is going to happen. Depending on the accuracy of their individual predictions, participants can either win or lose money, and thus have an incentive to be right. They should only participate if they think they have information that improves the current market forecast.

As in previous years, the PollyVote relied on the Iowa Electronic Markets’ (IEM) for predicting the popular vote. The IEM is operated by the Business School of the University of Iowa for teaching and research purposes. The market is relatively low volume and participants are not allowed to invest more than $500. The IEM is the only prediction market for forecasting the vote share, so one cannot combine across markets. We did, however, combine across time by calculating one-week averages to protect against short-term spikes.

Across the last six elections from 1992 to 2012, the IEM vote-share market was the second most accurate among the PollyVote’s components, after citizen forecasts. In 2016, however, the IEM performed poorly and provided the worst forecasts of all components throughout the campaign by wildly overestimated Clinton’s vote share.

Prior research shows that participants in these markets tend to be well educated and belong to middle and upper income groups. It may have been the case that this certain demographic group allowed their wishes to get the better of their judgment.

Expert judgment

Asking experts to predict what is going to happen is one of the oldest forecasting methods available. When it comes to forecasting elections, for example, experts have knowledge and expertise about how to read and interpret polls. In particular, experts know that polls have errors. Thus, they could be expected to be able to figure out in which direction the error is, and thus revise the forecast in the right direction.

In 2016, the combined expert forecast overestimated Clinton’s popular vote share relative to polls in five of the six surveys conducted since the parties’ conventions. On average, the expert forecast was 0.6 percentage points higher than the combined polls. In other words, the expert consensus was that the polls underestimestated Clinton’s support.

Citizen forecasts

As in previous elections, a forecast derived from survey respondents’ answer to a simple question “Who will win?” was among the most accurate methods for predicting the 2016 popular vote. Across the final 100 days before the election, citizen forecasts missed on average by only 1.4 percentage points, an error that is only slightly higher than the average across the six previous elections.

Models

The PollyVote used combined forecasts from two separate groups of models: index and econometric models.

Index models are based on the idea of prospective voting. That is, they rely on the idea that voters evaluate the candidates and how they stand on the issues when deciding upon for whom to vote. On average, the five available index models overestimated Clinton’s support, particularly due to two models that were especially far off (the bio-index and the issue-index). Three of these models (e.g., the big-issue model, the Issues and Leaders model, and the Keys to the White House) were close the final election outcome.

In contrast, most political-economic models are based on the idea of retrospective voting. That is, voters are expected to look back at how well the incumbent government has done its job, particularly in handling the economy. In addition, many models include some measure about how long the incumbent party has been in office to account for Americans’ desire for change, and some also include a measure of the president’s popularity. These  models predicted a very close race, at least on average, and thus were the most accurate component method this election, which is the first time that has happened since 1992. That said, the final forecasts from the 18 individual models differed by as much as 10-points, ranging from 44.0% to 53.9% of the two-party vote.

Combining forecasts

We know from prior research that the relative accuracy of the different methods varies from one election to the next, not just for election forecasting but also in other fields. We can see this again in 2016. Prediction markets, which were among the most accurate methods historically, were dramatically off, while econometric models, historically high in error, turned out to be more accurate this time. That is one of the reasons why combining forecasts usually works well. It’s extremely difficult to predict ex ante which method will end up being most accurate.

In 2016, five of the six component methods erred in the same direction, over-predicting Clinton’s vote share by an average of more than 2 percentage points. As already mentioned, the IEM, in particular, was way of the mark. The only component method that slightly underestimated Clinton’s support were econometric models. As a result, there was little “bracketing” of the true value, which limits the benefits of combining forecasts.

Given the little bracketing, the PollyVote performed only slightly better than the typical forecast. It performed worse than econometric models, citizen forecasts, and polls but outperformed expert judgment, index models, and prediction markets.

The principle of combining forecasts does not claim that the combined forecast will always be more accurate than any of its components. While that can happen – and did happen, for instance, in 2004 and 2012 – it is unlikely to happen in any single election.

The claim is that over time, as the relative accuracy of other methods varies, the combined forecast will outperform its components. This can be seen by looking at the mean absolute forecast errors across all seven elections, including this year’s. On average, the PollyVote error was lower than any of its components.

polly_pm2

Interestingly, citizen forecasts performed nearly as well as the PollyVote. So why not just use this one method in the future, you might ask? One major advantage of combining forecasts is that it’s often among the most accurate methods and, most importantly, it avoids large errors. There is not guarantee that citizen forecasts will perform as well in future elections.

How to improve

At the PollyVote, we are currently reviewing what we can learn from this election. This process includes reviewing which forecasts to include, how to better combine them, and how to better communicate their surrounding uncertainty.

The PollyVote aims at communicating scientific principles that can be used by any organization.

  1. We do know from prior research that the combined forecast will always be at least as accurate than the typical component forecasts, in any single election. So, it prevents you from making large errors.
  2. We also know that, over time and across many elections, the combined forecast will be among the most accurate forecasts available, because the performance of individual forecasts varies wildly.
  3. If you accept that it is extremely difficult to predict which forecast will turn out to be most accurate in a particular election, there is no better way to forecasting than combining forecasts.

The PollyVote forecasters are (in alphabetical order): J. Scott Armstrong, Alfred G. Cuzán, Andreas Graefe, and Randall J. Jones Jr.

Final PollyVote forecast: Clinton will win

All numbers are in and, according to the PollyVote, Hillary Clinton will become the first female president of the United States.

Popular and Electoral Vote Prediction

Clinton will win the popular vote by 5.0 percentage points in the two-party vote (52.5% vs. 47.5%). Based on the PollyVote’s historical error, Clinton’s chance to win the popular vote is above 99%. In terms of the Electoral College, Polly predicts Clinton to receive 323 electoral votes compared to 215 for Trump.

Forecast over the course of the campaign

Forecasting is useful only when it can aid decision-making. Thus, Election Eve forecasts are only for entertainment and betting— and writing victory and concession speeches.

The PollyVote has predicted Clinton to win beginning with its first release in early January 2016. Since the first Super Tuesday on March 1st, when it became clear that Trump would become the Republican nominee, Clinton’s lead in the PollyVote forecast averaged 5.6 points. Throughout the course of the campaign, it ranged from a minimum of 3.8 points to a maximum of 8.2 points. In other words, the PollyVote never had Trump close to winning.

Since its launch in 2004, the PollyVote has always correctly predicted the winner on any given day, months in advance, and with little forecast error. Tonight, we will find out whether Polly the Parrot will keep her clean slate. Check back in the days to come as we will provide analyses of the different forecasting methods’ accuracy.

Final expert survey: Clinton will win by 4.4 points

The PollyVote team has completed its 13th and final survey of elections experts to forecast the 2016 presidential election. In this survey, conducted between November 6 and 7, 12 academics from a variety of colleges and universities responded.

As in previous rounds, all respondents expect a Clinton win. However, her lead has narrowed again compared to the results of the previous survey conducted a week ago.

Whereas, in late-October, the experts expected that Clinton will win the popular vote by more than 5.5 points, the new average forecast is half a percentage point lower, at 52.2% of the two-party vote (or about a 4.4-point margin). The individual forecasts ranged from 51.0% to 53.3%, with a standard deviation of only 0.8 points.

Polly thanks the experts who participated in this round, namely

  1. Randall Adkins (University of Nebraska Omaha)
  2. Lonna Rae Atkeson (University of New Mexico)
  3. John Geer (Vanderbilt University)
  4. Sandy Maisel (Colby College)
  5. Michael Martinez (University of Florida)
  6. Thomas Patterson (Harvard University)
  7. Gerald Pomper (Rutgers University)
  8. David Redlawsk (University of Delaware)
  9. Larry Sabato (University of Virginia)
  10. Michael Tesler (University of California, Irvine)
  11. Charles Walcott (Virginia Tech)

and one expert who preferred to remain anonymous.

Clinton at new record high in PollyVote.com expert survey

The PollyVote team has completed its 11th survey of elections experts to forecast the national vote in the 2016 presidential election. In this survey, conducted between October 13 and 15, and thus after the second debate, 14 academics from a variety of colleges and universities responded.

As in the previous round, all respondents expect a Clinton win. However, her lead has widened substantially, to a new record high.

Whereas, in late September, the experts expected that Clinton will win the popular vote by 4 points, the new average forecast is 2.5 percentage points higher, at 54.7% of the two-party vote (or about a 9-point margin). The individual forecasts ranged from 53.3% to 57.4%, with a standard deviation of 1.4 points.

Polly thanks the experts who participated in this round, namely

  1. Randall Adkins (University of Nebraska Omaha)
  2. Lonna Rae Atkeson (University of New Mexico)
  3. John Coleman (University of Minnesota)
  4. George Edwards (Texas A&M University)
  5. Keith Gaddie (University of Oklahoma)
  6. John Geer (Vanderbilt University)
  7. Sandy Maisel (Colby College)
  8. Michael Martinez (University of Florida)
  9. Thomas Patterson (Harvard University)
  10. Gerald Pomper (Rutgers University)
  11. David Redlawsk (University of Delaware)
  12. Larry Sabato (University of Virginia)
  13. Michael Tesler (University of California, Irvine)
  14. Charles Walcott (Virginia Tech)

Experts predict Clinton will win popular vote by 4 points

The PollyVote team has completed its 10th survey of elections experts to forecast the 2016 presidential election. In this survey, conducted between September 28 and 30, and thus after the first presidential debate, 13 academics from a variety of colleges and universities responded.

Similar to the previous round, all respondents expect a Clinton win. However, her lead has narrowed substantially.

Whereas, in late August, the experts expected that Clinton will win the popular vote by 7 points, the new average forecast is 1.3 percentage points lower, at 52.2% of the two-party vote (or about a 4-point margin). The individual forecasts ranged from 51.6% to 53.7%, with a standard deviation of only 0.7 points.

 

Polly thanks the experts who participated in this round, namely

  1. Randall Adkins (University of Nebraska Omaha)
  2. Lonna Rae Atkeson (University of New Mexico)
  3. Keith Gaddie (University of Oklahoma)
  4. John Geer (Vanderbilt University)
  5. Sandy Maisel (Colby College)
  6. Michael Martinez (University of Florida)
  7. Thomas Patterson (Harvard University)
  8. Gerald Pomper (Rutgers University)
  9. David Redlawsk (University of Delaware)
  10. Larry Sabato (University of Virginia)
  11. Michael Tesler (University of California, Irvine)
  12. Charles Walcott (Virginia Tech)

and one expert who preferred to remain anonymous.

Bio-index model update: Clinton now at 58.3%

Variable 36 in Trump’s bio-index was coded incorrectly. Trump’s brother Freddy died from alcoholism in 1981, at age 41. Due to the variable recoding, the forecast has changed.

The updated vote-share forecasts are 58.3% for Clinton and 41.7% for Trump. This is the highest forecast for Clinton of all components aggregated in the PollyVote.

Clinton with 7-point lead in new expert survey

The PollyVote team has completed its 9th survey of elections experts to forecast the 2016 presidential election. In this survey, conducted between August 29 and 31, 13 academics from a variety of colleges and universities responded.

All respondents expected a Democratic win. The average forecast is that Democratic nominee Hillary Clinton will receive 53.5% of the two-party vote, with individual forecasts ranging from 52.1% to 55.8%. This value is the same as yesterday’s PollyVote prediction and 0.8 percentage points higher than the 52.7% for Clinton in the previous expert survey.

Polly thanks the experts who participated in this round, namely

  1. Randall Adkins (University of Nebraska Omaha)
  2. Lonna Rae Atkeson (University of New Mexico)
  3. John Coleman (University of Minnesota)
  4. George Edwards (Texas A&M University)
  5. John Geer (Vanderbilt University)
  6. Sandy Maisel (Colby College)
  7. Michael Martinez (University of Florida)
  8. Thomas Patterson (Harvard University)
  9. David Redlawsk (University of Delaware)
  10. Larry Sabato (University of Virginia)
  11. Michael Tesler (University of California, Irvine)
  12. Charles Walcott (Virginia Tech)

and one expert who preferred to remain anonymous.

Clinton leads by more than 5 points in new expert survey

The Pollyvote team has completed its 8th survey of elections experts to forecast the 2016 presidential election. In the late July survey, conducted between July 29 and 31, 12 academics from a variety of colleges and universities responded.

All but one responding experts expected a Democratic win. The average forecast is that Democratic nominee Hillary Clinton will receive 52.7% of the two-party vote, with individual forecasts ranging from 48.5% to 55.4%. The expert forecast is thus 1.5 percentage points lower than in the previous survey, when the mean forecast was that Clinton will receive 54.2%.

Polly thanks the experts who participated in this round, namely

  1. Randall Adkins (University of Nebraska Omaha)
  2. Lonna Rae Atkeson (University of New Mexico)
  3. John Coleman (University of Minnesota)
  4. George Edwards (Texas A&M University)
  5. Sandy Maisel (Colby College)
  6. Michael Martinez (University of Florida)
  7. Thomas Patterson (Harvard University)
  8. Gerald Pomper (Rutgers University)
  9. David Redlawsk (Rutgers University)
  10. Larry Sabato (University of Virginia)
  11. Michael Tesler (University of California, Irvine)
  12. Charles Walcott (Virginia Tech)

New high for Democrats in PollyVote expert survey

The Pollyvote team has completed its seventh survey of elections experts to forecast the 2016 presidential election. In the late June survey, conducted between June 28 and 30, 12 academics from a variety of colleges and universities responded.

All responding experts expected a Democratic win, with forecasts of the popular vote ranging from a minimum of 52.0% to a maximum of 57.0%. The mean forecast is that the Democrats will garner 54.2% of the major-party vote (compared to 45.8% for the Republicans).

The predicted Democratic vote share is thus 1.8 percentage points higher than in the previous survey, conducted in late May, when the mean expert experts had predicted a vote share of 52.4%. The current results also show the highest vote share for the Democrats in all expert surveys conducted since January.

Polly thanks the experts who participated in this round, namely

and one expert who preferred to remain anonymous.

Experts see Democrats’ lead narrowing to 5 points

The Pollyvote team has completed its sixth survey of elections experts to forecast the 2016 presidential election. In the late May survey, conducted between May 29 and 31, 13 academics from a variety of colleges and universities responded.

All but one responding experts expected a Democratic win. The average expert forecast for the Democratic share of the two-party vote was 52.4%, compared to 47.6% for the Republicans.

The predicted Democratic vote share is thus 0.8 percentage points lower than in the previous survey, conducted in late April, when the mean expert experts had predicted a vote share of 53.2%.

Polly thanks the experts who participated in this round, namely

and one expert who preferred to remain anonymous.