PredictWise Blog

Microsoft Prediction Lab's final probabilities

Bookmark and Share

Five weeks ago I launched a new website, with a few friends, including Miro Dudik and David Pennock, called Microsoft Prediction Lab. The website consolidates research into both non-representative polling and prediction games. I have spent years understanding how various raw data: polling, prediction markets, and social media and online data, can be transformed into indicators of present interest and sentiment, as well as predictions, of varying populations. Then, how decision makers allocate resources with the low latency and quantifiable market intelligence that we produce. Microsoft Prediction Lab allows us to continuously innovate not only on the path of raw data to analytics to consumption, but the collection of the data itself. Please take a look at this post on the mission of Microsoft Prediction Lab.

First, we would like the thank the thousands of active users who made this first game such an interesting and meanignful experience!

Microsoft Prediction Lab has a market for all 507 elections in the midterms: 36 senatorial, 36 gubernatorial, and 435 house. In each of these markets users can buy and sell contracts on the possible outcomes of each election. For example, in New Hampshire there are two possible outcomes: Democratic candidate Jeanne Shaheen and Republican candidate Scott Brown. A prediction on Shaheen would return 112 points for every 100 wagered, while a prediction on Brown would return 467 point for the same 100 points! If someone thought that Brown as undervalued, that there was a good return in wagering 100 points for 467, s/he should predict Brown and if s/he thought Shaheen was undervalued they should buy Shaheen. As people predict Brown, the return goes down and vice-versa.

The return that an investment settles on in a market is extremely correlated with the probability of the outcome. We show the translation of the price to the probability on the market alongside the return on prediction.

Further, the markets moved quite a bit over the last few weeks. Actually, there was movement in most of the 507 markets. In this market 85 people placed predictions, while others saw well into triple digits. To test how efficient that movement was, whether the crowd was supplying information, we captured the probabilities in all 507 races at about midnight on Election Eve.

These probabilities represent the probability of victory for the party just before 12:00 AM on Election Day. We look forward to checking back later this week to see how Microsoft Prediction Lab did ... and, for those of you playing the game, you have until 9 PM ET to keep the predictions coming!

Microsoft Prediction Lab - Final Probabilities

Democrats poised for (possible) great night … in governors elections

Bookmark and Share

The Democrats have a decent chance at having a great night in the governor’s races. There are 10 elections that I am following closely:

2 Tough, but possible pickups for the Democrats:

Wisconsin: this is a potentially a huge pick-up for the Democrats against a likely 2016 contender for the Republicans. Scott Walker is a conservative Republican leader in a solid blue state.

Maine: Paul LePage is a second unapologetic conservative Republican in a solid blue state. He is also in a tight spot.

3 Likely pickups for the Democrats:

Florida: neither the Democratic former governor Crist nor the current Republican governor Scott are pulling ahead in this nail-biter.

Alaska: independent Bill Walker (running with a Democratic lieutenant governor) is looking very strong against the Republican incumbent.

Kansas: Paul Davis is pulling ahead of conservative Republican Sam Brownback in this incredibly red state.

3 Likely holds for the Democrats:

Colorado, Illinois, and Connecticut: all three of these have looked tight, but are likely to remain blue governors in blue states.

1 Likely hold for the Republicans:

Michigan: this is a generally blue state, but there is a Republican incumbent heading towards a likely hold.

1 Likely loss for the Democrats:

Massachusetts: the only likely bad mark on the election for the Democrats is Martha Coakley. She is poised to lose against a moderate Republican Charlie Baker. Although, as a moderate Republican in a state with a long tradition of moderate Republican governors*, this is not a major ideological shift. *Note: I am referring to Governor Romney, a moderate, who should not be confused with Republican nominee for present Romney, who is not a moderate!)

Republicans at about 85% to capture U.S. Senate

Bookmark and Share

The Republicans are about 85% likely to take control of the U.S. Senate in January, 2015. This is going to happen, because the Democrats are going to win Blue States and the Republicans are going to win Red States. And, the Republicans are likely going to win two crucial Purple States (Iowa and Colorado). This election is not wave or a disaster for either party, but pretty much as should be expected. The most likely outcome is going to be the Democrats controlling 47-8 seats to the Republican 52-3.

The Democratic path to victory is very simple; they need to capture both New Hampshire and North Carolina, which are likely, and then three additional states of the five in play. The runoff system makes it very unlikely they will win in Georgie and Louisiana. And, I do not think Kansas’ Orman is going to make himself the swing vote with a 49 Democratic senate. Why would he do that if he can switch back in 2017 with seniority when the Democrats recapture the senate?

There is a possibility of a systematic polling bias against the Democrats (i.e., the Democrats will over-perform polls on Election Day). Nate Silver seems to think I am wrong (although the post is very long and does not address me directly, so it is hard to say for sure). He believes that the bias in 2012 is an historical anomaly. His argument is that states tend to break towards their fundamentals (i.e., Democratic states polls are biased against Democrats and Republican states polls are biased against Republicans). The problem is that he runs a regression with data from 1998 to 2012 and includes presidential polling along with senatorial. First, in 1997 the response rate for random digit dialing was still 36%, versus <9% in 2012. While I applaud the use of added observations, including time before the technical issues appeared for pollsters is a mistake that will smooth all treatment effects. Second, presidential polling is going to swamp senatorial polling. Not only in size, but in stability and accuracy. Presidential elections have more stable voter turnout.

Knowing there was an issue in 2012, I have no doubt that the pollsters will try to correct for any bias, but I believe it is likely that there will still be some bias against Democrats. First, with shifts in both coverage and non-response error moving fast it is not easy to correct for what happened in two or four years ago. Baseline data used for corrections is already obsolete (i.e., reliable lists of cell-phone only versus landline/cell phone are not updated fast enough!) and simply correcting for the error of four years ago is also obsolete (i.e., any correlations from four years ago are already wrong). This group of pollsters may under or over correct, but it is hard to see how they would be systematic unbiased. Further, they are technically very conservative people, so under-correcting is just more likely. Second, many pollsters are not knowledgeable enough to do that and will not bother. This group of pollsters are likely to favor Republicans.

Note: I had a very interesting early morning phone call today with Sam Wang of Princeton, so I expect he may touch on some of the same topics about bias today as well.

Any potential systematic polling bias, that favors the Democrats, will only get them Colorado and possibly Iowa; putting them at 49 seats. Colorado is most likely, because, with high population movement and Hispanic population, it is susceptible to bias. High population movement, young people with cell phones, leads to coverage error that favors Republicans. Hispanic population lead to non-response error, as Democratic Hispanics are less likely to answer English polls as Republican Hispanics. Iowa is the next most likely, as a Democratic leaning purple state shares some of the same attributes. It is less likely that Alaska or Georgia are going to be heavily hit by bias, as both states are much less susceptible. Kansas does not have a Democratic challenger!

Hidden Errors and Overconfident Pollsters

Bookmark and Share

Written with Sharad Goel and Houshmand Shirani-Mehr

Election forecasts, whether on HuffingtonPost's Pollster, New York Times’ Upshot, FiveThirtyEight, or PredictWise, report a margin of error of typically 3 percentage points. That means that 95% of the time the election outcome should lie within that interval. We find, however, that the true error is actually much larger than that, and moreover, polls historically understate support for Democratic candidates.

To estimate the true margin of error, we looked at all polls for senatorial races in 2012 that were published on the two major poll aggregation sites (Huffington Post’s Pollster and Real Clear Politics).Then, using the standard formula, we computed their theoretical margin of error. Finally, we simply plotted the percentage of polls where the outcome of the election actually fell within the standard confidence range.

Note: Data are from Huffington Post’s Pollster and Real Clear Politics. The thick horizontal line, a little below 70%, represents overall percent of outcomes in the reported 95% confidence ranges.

For polls conducted right before Election Day, the actual election outcome falls within a poll’s stated 95% confidence interval about 75% of the time. That means that whereas the polls’ margin of error says they should capture 95% of outcomes, they in fact capture only 75%. In other words, the reported margins of error are far too optimistic.

Why are the reported confidence intervals too narrow? First, polls only measure attitudes at the time they were conducted, not on Election Day, and the standard error estimates neglect to account for this. (To be fair, the pollsters typically add the disclaimer that results reflect the likely outcome of a hypothetical election held on the day of the poll.) But, close to Election Day, there is probably little real change in support, and the reported confidence intervals are still too small. This discrepancy is attributable to polling companies reporting only one of four major sources of error, as we describe below.

Sampling Error: This is the one source of error that pollsters do report, and it captures the error associated with only measuring opinions in a random sample of the population, as opposed to among all voters.

Coverage Error: Pollsters aim to contact each likely voter with equal probability and deviations from this result in coverage error. This was relatively easy in the world of ubiquitous landline phones (remember those), but with the rise of cell phones and internet it is not so easy to determine how to mix polling methods so that any given likely voter is contacted. This problem is getting worse each year, as landline penetration decreases. Coverage error is exacerbated by shifting modes of voting, such as voting by mail or early voting, which complicate traditional screens used to determine who is likely to vote.

Non-Response Error: After identifying a random set of likely voters, pollsters still need them to actual answer the polling questions. If those who are willing to be interviewed systematically differ from those who are not, this introduces yet another source of error, non-response error. This problem is also getting worse each year, as people are increasingly reluctant to answer cell-phone calls from unknown numbers or to take ten minutes to answer a poll in a busy world.

Survey Error: The exact wording of the questions, the order of the questions, the tone of the interviewer, and numerous other survey design factors all affect the result, leading to still another error source.

As Nate Cohn outlined in the New York Times on Thursday, the latter three error sources are more likely to undercount Democrats than Republicans. For example, Democrats are more likely than Republicans to have a cell-phone from a different area code than where they currently live (like all three of the authors of this article), which in turn results in coverage error since such individuals cannot be included in state-level polls. Cohn notes that among cell-phone only adults, people whose area code does not match where they live lean Democratic by 14 points, whereas those that matched lean Democratic by 8 points. For an example of non-response and survey error, Cohn notes that Hispanics who are uncomfortable taking a poll in English are more likely to vote Democratic than demographically similar Hispanics.

Thus, we expect the actual polling errors to be larger than the stated errors, and moreover, we expect polling results to favor the Republicans. This pattern is strikingly apparent when we plot the observed differences between poll predictions and actual election outcomes for the 2012 Senate races. Positive numbers indicate the poll skewed in favor of the Republicans. Alongside the observed differences, we plot the theoretical distribution of poll results if sampling error were the only factor.

Note: Data are from Huffington Post’s Pollster and Real Clear Politics.

The observed distribution clearly skews toward the Republican candidates. Further, the observed distribution is wider than the theoretical one, in large part because the polls are conducted over several weeks prior to the election, while the theoretical distribution does not take into account how much candidate support varies over the course of the campaign.

How much do these overly optimistic forecasts matter? First, the theoretical 3 percentage point margin of error is already substantial, and puts nearly every competitive race within that range. Second, when you add in the unaccounted for errors, election outcomes in contested races are simply far less certain; and coverage and non-response errors will likely only get worse each cycle. Third, while aggregating a bunch of polls for each election reduces the variance, it does not eliminate the bias, so these overconfident predictions pose a problem for aggregate forecasts as well. In short, those fancy models that show probability of victory are only as good as their ingredients, and if the polls are wrong, the poll aggregations will be wrong as well.

Sharad Goel is an Assistant Professor at Stanford University

David Rothschild is an economist at Microsoft Research and runs PredictWise

Houshmand Shirani-Mehr is a graduate student at Stanford University

Election Update - 11/2, 2 Days

Bookmark and Share

The Democrats are likely to lose the senate for two years. My predictions have been consistently more bullish on Republican victory than any of the other main forecasters: New York Times’ Upshot, FiveThirtyEight, HuffingtonPost’s Pollster, Princeton’s Sam Wang, etc. And, to be frank, the data is more generous to the Democrats than my gut, but I am obliged to run with the data.

The Democrats will have 47 seats if they take all of their certain races, along with New Hampshire and North Carolina. Of course, New Hampshire and North Carolina are not certain, but for the sake of this exercise, let us assume the Democrats take those seats. There are just eight other seats that are even remotely in play, and the Democrats would have to win three of them to get to a 50/50 tie, where Joe Biden is the tie-breaker.

Three of the races are extreme longshots: Arkansas, Kentucky, and Louisiana. In both Arkansas and Kentucky the Republican has been consistently leading by more than 5 percentage points. Neither of these states are particularly susceptible to polling error, they do not have fast moving populations, high levels of Hispanics, etc. So, it is unlikely they will suffer a catastrophic poling failure. In Louisiana Landrieu is within striking distance, but is hurt by the majority voting system. Senator Landrieu will not get 50% of the vote in the original vote and Democrats tend to suffer in runoffs, because Democratic voters are less likely than Republicans to bother voting twice. Winning any of these three elections is become extremely unlikely.

The Democrats really need to get three of the five other races, but they all pose their particular problems. First, Colorado is drifting back to the incumbent Democratic governor and, to be frank, the senatorial polling is a bit of mystery. The Democratic incumbent is liked and the state is reasonably blue. Despite consistent polling showing Udall losing, Colorado is a state that polling error is possible and early voting is confusing. Second, Iowa, like Colorado, is one that I would have expected the Democrats to challenge closer, but the polling is consistent for the Republican. This state is a little less blue than Colorado, less likely polling error, but the Democratic candidate has been closer all race. Third, Alaska is Republican state and the incumbent Democratic senator is polling consistently behind. He would not be in this race at all except for two crazy outlier polls showing him dominating. Fourth, Georgia shows the Republican in the lead and, again, the Democrats are not poised to do well in a possible runoff. Finally, the race in Kansas is a toss-up, but with the Republican governor almost definitely going to lose, expect people to split their vote in the ballot box and keep the Republican senator.

Actually, the Democrats really need to get three of the four race that are not Kansas. I doubt an independent Senator Orman will cast the deciding vote in the senate for the Democrats, because that would be political suicide for him in 2020. Instead, if the senate is 49 Democrats and 50 Republican, expect Orman to caucus with the Republicans in 2015-6 and then he will quietly caucus with the Democratic majority that will take over the senate on January 1, 2017.

All of this begs the question, can the Democrats capture Iowa, Colorado, and either Georgia or Alaska. It is possible, but if Sam Wang or Nate Silver were backing up their probabilities with real-money bets. To translate, Sam Wang is implicitly saying he is good getting $60 if the Democrats control the senate and paying me $40 if not, while Nate Silver is implicitly saying he is good getting $70 if the Democrats control the senate and paying me $30 if not. I consider a fair wager at $80 if the Democrats control the senate and $20 if not.

Here is New York Times and FiveThirtyEight compared with PredictWise. The one key difference is the other forecasters are much more bullish on the Democratic pickup in Georgia. I admit one key issue is that there is no historical identification for what will happen in a runoff that determines the balance of power in the U.S. Senate.

Updating Predictions: house, senatorial, senatorial balance of power, and gubernatorial.