PredictWise Blog

A response to David Leonhardt's "When the Crowd Isn't Wise"

Bookmark and Share

David Leonhardt wrote an excellent piece in the New York Times on both the effectiveness and limitations of prediction markets.

Mr. Leonhardt was inspired to write his column after prediction markets, notably Intrade, predicted that there was a 70 percent likelihood that the Supreme Court would overturn the Individual Mandate, which ultimately did not happen. This led to an explosion of criticism of Intrade on Twitter and elsewhere, with most commenters concluding that Intrade being "wrong" in this sample size of one proves that it isn't a useful tool for prediction.

We have a few points to add to this discussion, both in agreement with Mr. Leonhardt and in defense of markets.

First, we wish that Mr. Leonhardt had mentioned one of the primary advantages that prediction markets have over experts: while experts generally forecast a binary outcome for an event ("will happen" or "will not happen"), markets provide a probability of each outcome. Forecasting with probabilities versus binary outcomes provides a lot more information, but it is nuanced. For any given prediction, the only way that a market (or expert) can be easily understood as right or wrong is if it predicts with 100% certainty that an event will happen and then it does, or with 0% certainty that an event will happen and then it doesn't. This binary framework is easier for us to grasp, but it is not reflective of reality.

For those willing to consider the nuances, we can more accurately judge a probability forecast on two different rubrics: its calibration (i.e., percent that come true versus probability) and its precision (i.e., how far towards 0 or 100 percent it goes, while maintaining its calibration).

Calibration can be a tricky concept to grasp, but here's a simple example: If Intrade has markets for 10 different events and predicts a 90% likelihood for each of them to happen, 9 out of the 10 events should happen. Importantly, 1 out of the 10 should NOT happen. If all 10 happen, the markets were mis-calibrated, just as they were mis-calibrated if only 8 happen.

As for precision, as Mr. Leonhardt correctly pointed out, in times of low information the market should be less precise. This is certainly the case when it comes to a Supreme Court decision. To some extent, the market itself recognized that this was a difficult event to predict, and responded accordingly. That is why the markets stalled at 70 percent likelihood on the Supreme Court ruling rather than going up to 80 or 90 percent. The ex-ante information (and, to an extent, the ex-post information backs up the ex-ante) pointed to a likely overturn of the mandate, but the market was not willing to move into 'certain' territory. Further, the market also revealed a key indicator of low information on top of the low precision forecast. When the 'spread' of a contract - the difference between what people are willing to pay and what people are willing to sell for - moves past 5 percentage points, we consider the market to be illiquid. It did that prior to the Supreme Court ruling.

For markets with more information (such as the upcoming presidential election), we can expect far more precision, and as the sample size grows (such as the numerous markets for state-by-state Electoral College, senatorial, and gubenatorial elections) we can expect far better, and easier-to-understand, calibration.

We also wish that Mr. Leonhardt mentioned one other advantage: prediction markets update in real-time while experts update at will. Markets are always the most recent aggregation, and are available whenever you need the data. Providing easier access to this real-time data was one of the primary motivations for us to create PredictWise.

Mr. Leonhardt's conclusion, if we may paraphrase it, was dead on: prediction markets aren't perfect, especially for certain types of events, but they are still a very useful - perhaps the most useful - tool for prognostication. We agree with his suggestion that the future of predictions is to find innovative ways to combine the data gleaned from multiple sources (markets, polls, experts, etc.) to create the best possible predictions. That is our long-term goal at PredictWise.

In the coming weeks we will start providing state-by-state forecasts of the Electoral College, senatorial and gubernatorial elections and these predictions will reflect much more than just raw prediction market prices. The predictions will first debias prediction market prices and then combine them with forecasts derived from debiased-voter intention polling data and fundamental data. In theory, prediction markets should be perfectly efficient and should include all of the polling and fundamental data (i.e., they should not need any corrections or combine with other available data to form more accurate predictions). But, empirically, with extensive academic investigations, we know that is not the case.

Stay tuned.
-David & Andrew


June's tepid job growth, which led to no change in the unemployment rate, prompted the usual sour notes over for President Barack Obama's reelection chances. While the news is unquestionably bad for the president, the big picture is better for him than it may seem. In 1992, the economy added 379,000 jobs in the first half of the year. In 1996, when Clinton won reelection, the economy added 1,439,000 jobs in the first two quarters, and in 2004 it added 1,183,000 jobs in the same period. The economy has added 902,000 jobs in the first six months of 2012-not has high as Clinton or the younger Bush, but well over twice the growth during the first Bush's reelection campaign.

For the Obama campaign, this is a mixed blessing. While the President's economic numbers are not as bad as they may seem, his record is also largely set in stone. If history is any guide, neither a collapse or an economic miracle at this point will make much difference with an electorate whose mind is increasingly made up.

Click Here for the Full Text on Yahoo!'s The Signal

Prior to the ruling, there was no meaningful correlation between the healthcare decision and the presidential election. There was no discernible impact on Obama's reelection odds, for example, from the major shock in the likelihood of the Supreme Court ruling after the oral arguments. Now, we finally see that relationship: After the decision, Obama's odds of reelection ticked up a few points. The political markets-which are considerably more precise than the judicial pools-clearly see this a political victory for the White House, albeit a slight one that could evaporate quickly.

Click Here for the Full Text on Yahoo!'s The Signal

Many people have money riding on the Supreme Court's verdict on the Affordable Care Act, which is expected Thursday, from health care executives to 24-year-olds hoping to stay on their parents' insurance a few more years. But none have a more direct financial interest in the decision than those who have gambled on the outcome on The price on the site currently rests at a 75 percent chance that the court will overturn the individual mandate provision at the center of the case. When the court issues its decision, those who placed bets will find out if they won or lost-probably.

Situations like this test the strength of the online prediction markets in two ways. First, it can be tricky to define court decisions in a way that's precise enough to make it clear who won, particularly if the court takes a surgical approach to the law. Second, the crowd is less precise when it assesses the judgments of nine secretive people, compared to predicting the outcome of a popular election.

First, there's the difficulty in deciding what to bet on. The Affordable Healthcare Act consists of many different initiatives packed into one large law. At the core is the individual mandate requires that all Americans have healthcare or pay a tax. Several additional provisions stand out. The law requires insurers make insurance available to people with pre-existing conditions, expands Medicaid eligibility, sets up health insurance exchanges, and allows dependents to stay on their family's health insurance through their 26th birthday, to name a few. It would be interesting to see how the wonkiest gamblers predict the fate of any of these provisions. But it would be a nightmare to write a legal contract in a way that would make it clear who won and who lost under the almost infinite possibilities available to the court.

Click Here for the Full Text on Yahoo!'s The Signal

With the unrelenting focus on every aspect of the fight for the White House, it's easy to forget that the Senate is also up for grabs on Election Day. In fact, it could be the presidential election that determines the upper body's control: If the Republicans win a net total of three seats, the Senate will be divided 50-50. In that case, control will go to the party that wins the presidency. (If you need an eight-grade civics refresher, this is because one of the vice president's few official duties is to break a tie.)

Currently, the Democrats control 54 votes: 51 Democratic senators, two independent senators who caucus with them, and one vice-president. But the way the dice fell does not favor them: Democrats control 23 of the 33 Senate seats that are up this cycle, giving them much more territory to defend and many fewer opportunities to pick up seats. Of those 23 races, seven are open seats (i.e., the Democratic caucusing member is leaving the senate), while four of the 10 Republican seats are open.

The prediction markets are aware of this, of course, which is why the odds that Democrats will retain their majority currently rest at 41.5 percent. That's a major improvement for them since the beginning of the year, when their odds clocked in at 25 percent.

Click Here for the Full Text on Yahoo!'s The Signal