Polls and pollsters became an interesting sidebar during the presidential election. Nate Silver’s FiveThirtyEight blog at the New York Times became a topic of controversy. Conservatives vilified him as a stooge for the Obama campaign. Liberals vilified conservatives for living in denial about scientifically determined predictions. In many ways, I heard shades of the climate change meme playing out.
I haven’t studied Silver’s methods in detail. Using a variety of screens, he weighted the significance of various polls. He looked at past trends. He published odds of outcomes for the presidential and Senate races based on his analysis. I read him from time to time, and found him to be an interesting read; a real data geek’s delight. He predicted the correct outcome for the presidential race in every state and for 32 of 33 Senate races. Game, set, and match? Silver is a science genius. Right?
Physics Central blogger, Buzz Skyline, says not so fast. There is something fishy in the numbers. In his post How did Nate Silver Get the Election Odds so Wrong? he writes:
Bear in mind that I'm not saying Silver was wrong about who would win the election. If anything he was more right than his own numbers said he should have been. And being more right than he should be means there's something odd, or interesting, about his statistics.
Skyline uses a very helpful analogy. Imagine a weather forecaster predicting the chances of rain each day for a week. She gives the following chances of rain:
Monday
0.846
Tuesday 0.794
Wednesday 0.843
Thursday
0.906
Friday
0.797
Saturday
0.967
Sunday
0.503
What are the chances that it would have rained every day of that week? We get the answer by multiplying each of these percentages:
Probability = 0.846*0.794*0.843*0.906*0.797*0.967*0.503
When you do the math:
Probability = 19.9%.
What Skyline points out is that, instead of temperatures, these percentages were probabilities of an Obama win that Silver calculated for seven swing states:
New Hampshire
0.846
Virginia
0.794
Iowa
0.843
Ohio
0.906
Colorado
0.797
Wisconsin
0.967
Florida
0.503
If you put all the swing states into this calculation, using Silver’s percentages for the probable winner in each state, then the chances of predicting all of them correctly is 12%. The chance of him getting all the state presidential races correct and getting all but one of the Senate races correct are 6%. So either Silver is incredibly lucky or something else is going on here.
Furthermore, if you used the aggregation of polls done by Real Clear Politics you get nearly the same result. RCP missed Florida, the state Silver said had a 50.3% for Obama, a toss-up in other words. RCP was off on two Senate races instead of one. There were other pundits who predicted the same electoral vote count Silver did, just by looking at the polls.
Skyline writes:
In any case, it at least looks to me like the odds Silver published are probably too low and should in fact have given Obama and Romney much higher probabilities of carrying the states they each won, and therefor should have implied a much higher likelihood of Obama taking a second term as president.
And:
I'm not going to get over my discomfort until I understand why Silver is so good at predicting the outcomes of elections but apparently so bad at reporting the actual odds that are supposed to provide the predictions.
Read Skyline’s whole piece. It seems likely that the real winners in this debate were the pollseters themselves, not Silver. The pollsters got it remarkably right. With Skyline, I don’t think there was anything deceitful going on here but I do suspect there may be confirmation bias at work, seeing “science” where it may not have indeed existed. More answers are needed.
I find Skyline’s reasoning persuasive. What do you think?
