Oscar Night 2015 - Why I am stoked PredictWise got 20 of 24 (not 24 of 24) Oscar predictions “right”
DavidMRothschild on February 25, 2015 @ 2:14PM
Our Oscar predictions have been 19 for 24, 21 for 24, and 20 for 24 over the last three years, in the binary outcome space (i.e., the most likely candidate won the Oscar). Of the 12 “misses” 11 have been the second most likely and one has been the third most likely. But, our predictions are not probabilities for a reason; if we only cared about which candidate was the most likely and not how likely, we would not bother calibrating the difference!
What we are most proud of is the calibration of the Oscar predictions. In the 72 categories (24 per year) we have forecasted in the last three years, the average forecast for the leading candidate was 82%. Thus, on average, we expected to “win” a category 82% of the time and “lose” a category 18% of the time. Thus, 0.82*72 = 59 “wins” and 0.18*72 = 13 “losses” in expectation. Our 60 “wins” is pretty well calibrated!
A better way to think about calibration is to look at the 365 predictions we have made in the last three years. Of course, the predictions are not independent of each other (only one candidate can win in any category/year combination), but with five candidates in a category (except up to 9 in Picture and 3 in Makeup and Hairstyling) it is reasonable to use all predictions in testing calibration. On the x-axis we round each prediction into six buckets and on the y-axis we plot the percentage of predictions in that bucket that actually occur.
In an ideally calibrated set of predictions, the answers would all lie on the 45 degree line; if the average prediction is 20% in a group of predictions, it should occur 20% of the time. All three years are extremely well calibrated.
The final Oscar predictions are prediction market-based; the model is in this paper.
DavidMRothschild on February 22, 2015 @ 12:49PM
12:06 AM ET: 20 for 24 with Birdman's win!!! Another huge victory for prediction markets. Perfectly calibrated with average 82% for most likely candidate in the 24 catagories = expected outcome of 20 wins. And, a prediction markets did a great job in raising probability of victory of Birdman to 90% as Oscar night unfolded ...
11:56 PM ET: With Actor Eddie Redmayne (The Theory of Everything) and Actress Julianne Moore (Still Alice) both winning, we are 19 for 23 with just one to go!
11:43 PM ET: The markets were increasingly convinced as the night progressed and it happend ... Birdman's Alexandro G Inarritu take the Director award! Brings us 17 of 21 with just 3 to go ...
11:35 PM ET: OK - now 16 of 20 with the Imitation Game's win for Adapted Screenplay, but another tight loss for Original Screenplay where had Grand Budapest Hotel, but Birdman second ...
11:25 PM ET: Grand Budapest Hotel for Original Score (our third loss, all reasonably probable second place choices). That brings us to 15 of 18 … now for Adapted Screenplay, Original Screenplay, Actress, Actor, Director, and Picture!
11:10 PM ET: 15 for 17 with Selma's Glory winning Song ...
11:00 PM ET: All the big awards coming up now ... interesting note is that the markets are coalescing around Birdman's Alexandro G Inarritu for Director. Boyhood's somewhat surprising loss in Film Editing could be seen as a pre-curser ...
10:51 PM ET: CitizenFour gets us back on track for Documentary Feature ... 14 of 16 overall ...
10:44 PM ET: OK - another second favorite taking it ... with Whiplash taking the Film Editing catagory. That closes out the technical catagories at 7 for 8 ... 13 for 15 overall!
10:30 PM ET: Reminder, I am highlighting the winners here based on our final Oscar predictions from 1 PM ET (13 for 14)
10:26 PM ET: And Birdman's win for Cinematography makes 13 for 14 ...
10:23 PM ET: Back on track with Grand Budapest's 3rd win in Production Design ... 12 for 13 ...
10:13 PM ET: In the crazy animation catagory, seriously no Lego Movie, we sadly fall to 11 for 12 ...
10:08 PM ET: Stoked for being 11 for 11, but amazed at being 3 for 3 in the shorts. Markets rock at aggregating information!!!
10:04 PM ET: 10 for 10 with Interstellar with their Visual Effects.
9:54 PM ET: 9 for 9 after Patricia Arquette's win for Boyhood.
9:50 PM ET: 8 for 8 is great start! Just nailed the two sound catagories with Whiplash for Mixing and American Sniper for Editing.
9:40 PM ET: Keeping the list updated here (6 for 6)
9:30 PM ET: 6 for 6, but more important 2 for 2 so far in the 3 crazy hard "short film" catagories. Congrats to Crisis Hotline Vets for best documentary short.
9:26 PM ET: 5 for 5, but more important 1 for 1 so far in the 3 crazy hard "short film" catagories. Congrats to The Phone Call for best live action short.
9:11 PM ET: Ida as foreign film brings us to 4 for 4 ... upcoming are the very difficult short films!
9:01 PM ET: Grand Budapest takes its second with makeup and hairstyling ... we ar 3 for 3.
8:58 PM ET: We are 2 for 2 with Grand Budapest Hotel winning its first for costume design. It is favored in a leading 4 catagories.
8:40 PM ET: We are live and the first Oscar is for Supporting Actor ... phew 1 for 1 ...
1 PM ET: Here are the final Oscar predictions. A few things to consider:
1) I rounded to the nearest percent. Nothing is 100%; anything that says 100% is just 99.5% or greater.
2) Market-based forecasts have been binary correct for 19 and 21 of 24 in the last two years, I expect 20 of 24.
3) If you want to win your pool tonight just pick all of the favorites. If there are many people in the pool or the pool has some other people reading this column, switch in one or two second-most likely that you want to win.
4) Birdman is favored in two key categories (Director and Picture), but nothing else. Boyhood is favored in two categories, but second in both Director and Picture. Grand Budapest if favored in four categories!
5) The prediction page will update in real-time as the event unfolds (as early awards could add information about later awards). I will blog about anything interesting on this page …
DavidMRothschild on January 12, 2015 @ 7:03AM
As promised we got 8 of 10 major movie categories binary correct at the Golden Globes. Boyhood took home its three statues, but Birdman came up short with 2 of 3 excepted wins. The surprise winners, The Grand Budapest Hotel for Best Picture (comedy or musical) and Amy Adams in Big Eyes for Best Actress (comedy or musical), were both our second most likely choices.
DavidMRothschild on January 09, 2015 @ 5:54PM
The Golden Globes are Sunday, January 11 and, as usual, we have some bold, market-based, predictions on PredictWise. The methodology for the Golden Globes is the exact same as the Oscars, which has not been previously tested out-of-sample for the Golden Globes, but we are excited to see how they do. We have predictions for 10 categories and here are our pre-event predictions:
We have the drama Boyhood looking very strong in Best Director, Best Picture (drama), and Patricia Arquette as Best Supporting Actress. We have the comedy Birdman rivaling Boyhood with a favored position in Best Picture (comedy or musical), Michael Keaton as Best Actor (comedy or musical), and Best Screenplay. Eddie Remayne leads the Best Actor (drama) category from The Theory of Everything, Julianne Moore from Sill Alice as the Best Actress (drama), Emily Blunt from Into the Woods as Best Actress (comedy or musical), and J.K. Simmons from Whiplash as Best Supporting Actor. The closest categories are Best Actress (comedy or musical) with Amy Adams from Big Eyes in a close second, and Best Screenplay with The Brand Budapest Hotel and Boyhood both challenging.
We expect to get 8 out of 10 correct with an average probability for the leading nominee at 82%. Enjoy the show and follow the live predictions here: http://www.predictwise.com/GG2015.