DavidMRothschild on March 02, 2014 @ 8:17PM
12:35 PM ET: My final thought for the night is that this is another case of pundits and insiders advertising a close event when the proper aggregation of data said it was not. As I noted on Twitter earlier, my acceptance speech is short. I would like to thank prediction markets for efficiently aggregating dispersed and idiosyncratic data. And, thank you to Miro Dudik and Deepak Pathak for crunching the data with me on the paper that determined the methodology for these predictions. And, thank you to Andrew Leonard and David Pennock for helping to make PredictWise a destination for data-driven prognostication! Good night … and, one last time, 21 for 24!!!
12:15 PM ET: There are two ways to judge accuracy. First, and most obvious, you want a small error. Correctly predicting 21 out of 24 categories and having non-negligible probability for the three other winners is indicative of a small error. Second, you want the predictions to be well calibrated. The top nominee in each category averaged 86.5% likelihood of victory. Thus, multiplied out over 24 categories we expected to pick the winner 20.76 categories. Winning 21 out of 24 means that our predictions were perfectly calibrated. If we picked any more categories we should have had higher probabilities of victory (i.e., if you get 24 out of 24, you should have 100% likelihood for the winner in each of the categories) and if we picked any less we should have had lower probabilities.
12:00 AM ET: 21 for 24.
11:54 PM ET: With the leading actress and actor categories we are up to 20 out of 23 with just the picture award left.
11:40 PM ET: 12 Years a Slave is now 7 percentage points more likely to win Picture since start of the show. Currenly 18 for 21 with 3 awards left!
11:30 PM ET: Excited about original screenplay ... that was the tightest of all of he categories! Now we are 17 for 20 with 4 awards left!
11:19 PM ET: Got the two music categories right; Gravity for score and Let it Go for song ... now 15 for 18 and breathing a little easier ... 6 more awards to go!
10:48 PM ET: Now 13 for 16, perfect in the technical categories … getting stoked for the big awards!
10:37 PM ET: Now 12 for 15 … just one more of the less prominent categories and then the big ones!
10:11 PM ET: Up to 9 for 12 after Gravity's double win in sound mixing and editing.
9:40 PM ET: Rough patch, now 6 for 9. 1 for 3 on short films.
9:29 PM ET: Now 5 for 6. Missed on Animated Short Film with surprise win by Mr. Hublot, but big wins with Frozen for Animated Feature Film and Gravity for Visual Effects.
9:02 PM ET: Two less prominent categories go our way, another Oscar for Dallas Buyers Club in Makeup and Hairstyling and an Oscar for The Great Gatsby in Costume Design.
8:43 PM ET: They started with a pretty secure category, best supporting actor ... stoked to start out 1 for 1 though ... phew!
8:20 PM ET: I am on the couch, I got a couple of beers, and I am looking forward to a fun Oscar night. I will update live here all night long, so stay tuned!
DavidMRothschild on March 01, 2014 @ 3:02PM
I always emphasis four traits to any good indictor or prediction: accurate, relevant, timely, and scalable. Accurate means a small error and well calibrated. Relevant means that the values the right value for the stakeholders. Timely means that the values starts early and update regularly. Scalable means that the methods and platforms allow many different values to be generated with minimal marginal cost.
The Oscar predictions at PredictWise are all four. First, we offer the probability of victory for all 24 categories. While some people offer binary outcomes and some people offer 6 or 8 or 9 categories, we do not settle for anything short of the right question for the right outcomes. Second, we are confident that our predictions are well calibrated in that if we say 10 things are 90% to happen, 9 of them will happen. Right now we have about 85% likelihood on the leader for all 24 categories. That means we hope that between 20 and 21 are pointing towards the winner. Third, our predictions started just after the nominations and have moved with the results of the awards season or other outside events. Fourth, our scalability is what allows us to jump into the Oscars as easily as we will jump into March Madness and the World Cup this summer.
So, if you are scoring at home, here are our predictions as of late on February 28, 2014. The predictions will change during the lead-up to the awards show and the show itself. I will be live blogging during the Oscars on this site and tweeting updates. Live values are here, here, and here:
DavidMRothschild on February 21, 2014 @ 10:13AM
Over the summer I got together with my research assistant Deepak Pathak and my colleague Miro Dudik to take a look at four different types of Oscar data: fundamentals, polling, prediction markets, and experts. While there are certainly meaningful things to learn from all of the data sources, properly translated prediction market prices were, by far, the most superior data source for creating continuously updating and accurate forecasts for all 24 Oscar categories.
Where did the other data go wrong?
Forecasts created with fundamentals (i.e., box office receipts, number of screens, release dates, other awards shows, etc.) are simply not that accurate across 24 categories. While this data is enticing, because it does so well in sports and politics, when you think a lot about it (like we did) you realize the huge problem; this data does not provide much identification between categories like song or makeup. If a movie does well in the box office, is because people like the song or the makeup? Further, there needs to be separate model for each one of these categories. While the same variables correlate for any football game or any of the Electoral College elections in a given year, the song or makeup categories all need different variables.
Polling has the potential to create both accurate and timely forecasts, but it requires incentives for frequent responses by high-information users to stay timely, and proper transformation of raw polls into forecasts to be accurate. By definition, real money prediction markets provide the incentives that polls generally lack.
Experts can create something similar to fundamental models, but it is not clear ex-ante which experts are going to work as most of the methodology is opaque.
That leaves prediction market data; the prices that investors are willing to sell contracts that will be worth either $1 or $0 depending on whether the outcome occurs or not. The price is highly suggestive of a probability, but theory, backed by empirical data does allow us to translate raw prediction market prices into probabilities.
The paper, which suggests a method derived from our review of these four data types, is currently under academic review, but you can see how its method does in 2014 on here (and here and here); I launched the predictions shortly after the nomination and they have been updating every few minutes since then. Here are the current predictions for eight big categories. They are scary confident with an average of 84%.
Best Picture: 12 Years a Slave is at 84%, which is a strong lead over Gravity at 14%.
Best Director: Gravity’s Alfonso Curaon at 98% is dominating this race with 12 Years a Slave’s Steve McQueen at just 2%.
Best Leading Actor: Dallas Buyers Club’s Matthew McConaughey at 90% is leading big over The Wolf on Wall Street’s Leonardo DiCaprio at 7%.
Best Leading Actress: Blue Jasmine’s Cate Blanchett at 98% is a massive favorite over American Hustle’s Amy Adams at 2%.
Best Supporting Actor: Dallas Buyers Club’s Jared Leto at 97% is also a strong favorite over Captain Phillip’s Barkhad Abdi at 2%.
Best Supporting Actress: In one of the tightest categories I have, 12 Years Slave’s Lupta Nyongo at 62% is leading American Hustle’s Jennifer Lawrence at 37%.
Best Adapted Screenplay: 12 Years a Slave at 88% is a strong favorite over Philomena at 7%.
Best Original Screenplay: In the only real-toss up of the top categories I have Her at 53% to American Hustle at 42%.
As a disclaimer, I have only seen two of the movies mentioned in all of these categories, but I am not saying which ones!
DavidMRothschild on September 26, 2013 @ 9:01AM
We obsess about the aggregated prices that emerge from markets, whether it is oil, the Dow Jones, or the prediction market contract on who will be the next president of the United States. The price is a reflection of the subjective beliefs of individual traders, and we spend too little time considering the individual traders’ expectations, strategies, and motivations that combine to create that price. Rajiv Sethi of Barnard College and I were very lucky to examine a unique dataset this summer, which allowed us to learn more about how individual traders behave in markets; specifically, we examined trade-level data for all trades that occurred in the final two weeks of the 2012 election for either Obama or Romney to win on Intrade, the largest political prediction market in 2012. Our main academic finding is that traders are surprisingly one directional, almost always buying contracts favoring one of the two candidates. A secondary finding that has garnered popular attention is that one trader heavily influenced the price of these contracts, possibly for potential political gain, by investing nearly $4 million in Romney positions over two weeks.
The academic question our paper examines is the trading strategies and motivations of the traders. And our finding is that most traders are either performing arbitrages (i.e., buying and selling contracts that guarantee them a small return) or trading in one direction (i.e., only going long on Obama or Romney). The archetypical informational trader gets new information and goes long on whichever side it favors. What we see is that traders get new information and use it to keep trading for their chosen candidate. An example would be a situation where there was bad news for the Romney campaign. Obama traders would go long for Obama and push up the price for Obama, but after a little while, the Romney traders would push back when they thought the price has moved too far towards for Obama. Everyone gets to keep going long for their candidate, but the new price still reflects the new information. Rajiv goes into this in more detail in an earlier blog post.
But, you are probably not reading this article to learn about trading strategies, so I will pivot to the possible market manipulation.
Many observers noted in real-time that a peculiar wedge opened up between Intrade and another prediction market, Betfair. Intrade was consistently 5-10 percentage points more bullish on Romney (i.e., a contract that paid out $1.00 if Obama won could trade for $0.70 on Intrade and $0.75 or $0.80 on Betfair). Our new dataset shows that one trader was making that happen by providing massive amounts of liquidity to the Romney side of all trades (accounting for roughly 1/3 of all action favoring Romney) and creating “fire-walls” of several hundred thousand dollars to keep the Romney price static at times of high information flow (e.g., if the price for an Obama contract was $0.70 the trader would note on the order book the willingness to sell $100,000 or more of Obama to win at $0.70 so that Obama traders could keep buying at that $0.70 price indefinitely without the price going up). Note that we have no personal information on this trader, just the trader’s trades.
We provide three possible explanations for the trader’s strategy: (1) the trader was convinced that Romney was underpriced throughout the period and was expressing a price view, (2) the trader was hedging an exposure held elsewhere, or (3) the trader was attempting to distort prices in the market for some other purpose.
(1) Simply going long Romney is unlikely because at any point the trader could have gone to Betfair and purchased the same contract for less money. There are many costs to utilizing Betfair: it trades in British Pounds, it blocks U.S. ip addresses, etc. But, a trader of this size could have overcome those costs at less money than the trader’s loss by utilizing Intrade versus Betfair.
(2) Hedging is a more plausible explanation for the trader’s behavior, but still unlikely. Earlier academic literature shows that some market indexes and the likelihood of the election outcome can be correlated, but we could not find similar patterns in 2012 (e.g., historically, sudden increases in the likelihood of a Democratic victory are adversely correlated with the S&P, despite historical correlations of stronger stock markets under Democratic presidencies). We cannot eliminate the possibility that the trader was hedging some more detailed securities like specific energy contracts that traders may have concluded would be more tightly impacted by the election outcome (e.g., the trader could have been going long Romney to cover potential losses in renewable energy contacts should Romney have won the election).
(3) Distorting the price for some other purpose, possibility political, is the most likely motivation of the trader. Placing hundreds of thousands of dollars on the order book, precisely at times of a lot of new information, is a strategy that maximizes the impact of the investment, but not the return. One of the trader’s most active periods was between 7:30 PM and 9:00 PM ET on Election Day, when new information was arriving by the second. At that time the trader essentially placed so many potential trades at about $0.70 per $1.00 for Obama that the trader was telling the market that s/he would match anyone who wanted to buy Obama at that price. This froze the price for the crucial 1.5 hours between the first major reports of election returns and the last swing state poll closings. Ultimately, the trader spent about $375,000 during that 1.5 hours and, as soon as the trader left the market at 9:00 PM, Obama shot up past $0.90 per $1.00 contract.
If this trader were attempting to manipulate the market, it is a three step process to make it successful: (1) change price, (2) convince people it is real, and (3) have people change behavior because of it.
(1) One trader was able to control the price of a liquid market with a massive wall of limit orders for two reasons. First, markets, by design move with large quantities of money regardless of whether it is 1,000 people with $100 or 1 person with $100,000. A sea of traders could move a stock price or Warren Buffett alone could move a stock price, as the market does not care about the motive or quantity of traders. Second, the government’s harassment of Intrade, and other online markets, made it difficult and risky to join and keep money in Intrade, which limited participants and readily available money. Thus, even if it was tempting to buy Obama long at $0.70 per $1.00 contract on Election Day, it is likely that few traders had the money sitting in the market necessary to buy up all of the contracts that the Romney trader was willing to sell. You would need to already be a trader and have tens or hundreds of thousands of dollars sitting idle in your account.
(2) This is the hardest step; it is not as easy to convince people that the price level is real, but maybe people who want evidence will appreciate any data source that validates them. Sites that present prediction market data, frequently aggregated Intrade with other markets to ameliorate the concern that any one data source can be wrong. Overall, prediction market data was very successful in providing accurate and timely predictions of the 2012 election. Yet, if someone was looking for a reason to be hopeful about Romney, Intrade’s price provided a solid piece of data for them.
(3) If people are convinced it is real, the impact on the campaign is going to happen. There is a cascading effect to being a viable candidate; the more viable a candidate appears, the more money and volunteers, support, and turnout the candidate receives. Thus, the more viable the candidate appears, the more viable the candidate becomes. When it comes to Election Day, one piece of positive news may be the validation someone needs to stop off at the polling place on the way home from a long day at work.
If manipulation could be successful, it would be worth it. If a few million dollars could boost fundraising and morale, than it would be a good investment next to one more television advertisement in a flooded Ohio, Florida, or Virginia market. Roughly $28 million was spent on TV advertising in just one state, Ohio, in the last week of the election alone.
We cannot say for sure if this market was manipulated, but someone definitely shifted the price heavily towards Romney and maintained that price imbalance until 9 PM on Election Day, when the polls closed in the last swing state and the election was finally up to the vote counters. Yet, despite this trader’s efforts, most observers, even if they were following just prediction markets, still received a very accurate forecast of the election.
This column syndicates with the HuffingtonPost.
DavidMRothschild on September 04, 2013 @ 8:05AM
On August 4 I tweeted that “Smart money is on de Blasio edging out [Bill] Thompson” for the second spot on the runoff. I followed that up by noting that Christine Quinn’s trajectory was troubling; it is not a good sign for a runoff if you are heading in the wrong direction. Both statements proved prescient as de Blasio was fourth in the polls on August 4 and is the current heavy favorite to be the next mayor of New York City. Meanwhile Quinn’s downward trajectory may push her out of a potential runoff, or even ameliorate the need for a runoff. But, Twitter does lead a little too much to the imagination, so here are some more details on the New York City mayoral contest.
The election in New York City is, potentially, a three step process. First, both parties have a primary on September 10. Second, if no candidate receives over 40% of the vote in the primary, there is a runoff between the top two candidates on October 1. Third, there is an election between the Democratic and Republican candidate on Tuesday, November 5.
Through the end of July Quinn and Anthony Weiner were trading spots on the top of the polls with de Blasio and Thompson battling it out for third and fourth. The New York Times poll that was in the field from August 2-7 showed the completion of Weiner’s fall to nearly single digits, but Quinn still had a lead and there were a huge amount of undecided voters; de Blasio was third with 14% and Thompson second with 16%. After that, August saw a string of six straight polls with de Blasio leading, finally blowing past 40% with the latest Quinnipiac poll that was in the field from August 28-September 1.
Meanwhile, Quinn continued to plateau through July with a string of polls with her leading, but always in the 22% to 32% range and no upward trajectory. As the frontrunner, this was troubling, because she has a lot of time to make her case to the Democratic electorate. Since then she has shown a consistent downward trajectory with the last three polls putting her below 20% and, crucially, below Thompson.
The smart money I was referring to on August 4 was the bookies like Paddy Power, Stan James and few other others; but, it was not as easy picking out the best odds. First, Quinn had the best odds of all, so she was still the favorite to get one of two spots in the potential run. Second, de Blasio had slightly more favorable odds than Thompson. The polls showed them nearly tied, but the betters favored de Blasio. Thus, the smart money had de Blasio pulling ahead of Thompson. Third, with Quinn still dominating the polls the bookies had no reason to push Quinn down below de Balsio and Thompson; it was still safe money to assume she had a higher probability than either de Blasio or Thompson. But, did she have a higher probability than the ultimate winner of the "not-Quinn" fight between de Blasio and Thompson; in my reading of the data, no.
The likely Republican candidate, Joe Lhota, is trailing any Democratic candidate by a wide margin. After five terms of Republican mayors, the Big Apple looks poised to put a Democrat back in Gracie Mansion and it is about 60-65% Bill de Blasio will be the next mayor of NYC.1
Then again, my friend Andrew Gelman of Columbia has a timely reminder for us in his blog: primary elections are hard to predict.