Nate Silver - He Blinded Us With Science?

Polls and pollsters became an interesting sidebar during the presidential election. Nate Silver’s FiveThirtyEight blog at the New York Times became a topic of controversy. Conservatives vilified him as a stooge for the Obama campaign. Liberals vilified conservatives for living in denial about scientifically determined predictions. In many ways, I heard shades of the climate change meme playing out.

I haven’t studied Silver’s methods in detail. Using a variety of screens, he weighted the significance of various polls. He looked at past trends. He published odds of outcomes for the presidential and Senate races based on his analysis. I read him from time to time, and found him to be an interesting read; a real data geek’s delight. He predicted the correct outcome for the presidential race in every state and for 32 of 33 Senate races. Game, set, and match? Silver is a science genius. Right?

Physics Central blogger, Buzz Skyline, says not so fast. There is something fishy in the numbers. In his post How did Nate Silver Get the Election Odds so Wrong? he writes:

Bear in mind that I'm not saying Silver was wrong about who would win the election. If anything he was more right than his own numbers said he should have been. And being more right than he should be means there's something odd, or interesting, about his statistics.

Skyline uses a very helpful analogy. Imagine a weather forecaster predicting the chances of rain each day for a week. She gives the following chances of rain:

Monday          0.846
Tuesday          0.794
Wednesday     0.843
Thursday        0.906
Friday             0.797
Saturday         0.967
Sunday           0.503

What are the chances that it would have rained every day of that week? We get the answer by multiplying each of these percentages:

 Probability = 0.846*0.794*0.843*0.906*0.797*0.967*0.503

When you do the math:

Probability = 19.9%.

What Skyline points out is that, instead of temperatures, these percentages were probabilities of an Obama win that Silver calculated for seven swing states:

New Hampshire           0.846
Virginia                        0.794
Iowa                             0.843
Ohio                             0.906
Colorado                      0.797
Wisconsin                    0.967
Florida                         0.503

If you put all the swing states into this calculation, using Silver’s percentages for the probable winner in each state, then the chances of predicting all of them correctly is 12%. The chance of him getting all the state presidential races correct and getting all but one of the Senate races correct are 6%. So either Silver is incredibly lucky or something else is going on here.

Furthermore, if you used the aggregation of polls done by Real Clear Politics you get nearly the same result. RCP missed Florida, the state Silver said had a 50.3% for Obama, a toss-up in other words. RCP was off on two Senate races instead of one. There were other pundits who predicted the same electoral vote count Silver did, just by looking at the polls.

Skyline writes:

In any case, it at least looks to me like the odds Silver published are probably too low and should in fact have given Obama and Romney much higher probabilities of carrying the states they each won, and therefor should have implied a much higher likelihood of Obama taking a second term as president.

And:

I'm not going to get over my discomfort until I understand why Silver is so good at predicting the outcomes of elections but apparently so bad at reporting the actual odds that are supposed to provide the predictions.

Read Skyline’s whole piece. It seems likely that the real winners in this debate were the pollseters themselves, not Silver. The pollsters got it remarkably right. With Skyline, I don’t think there was anything deceitful going on here but I do suspect there may be confirmation bias at work, seeing “science” where it may not have indeed existed. More answers are needed.

I find Skyline’s reasoning persuasive. What do you think?

Comments

[Mike Aubrey]

I think you're not giving Silver enough credit. His model is more complex than that. And this analysis ignores the fact that he was very nearly just as accurate in his predictions in 2008. It also ignores the fact that he's also be incredibly successful with his numbers in sports predictions and gambling.

The critique sounds nice. It sounds convincing. But only if the only thing Silver has been successful with was this 2012 election. Silver himself has emphasized that the polls are really accurate.

The weather analogy is an interesting one. Particularly because the description of it demonstrates that its flawed by unnecessarily simple assumptions about weather and that correlates directly with the unnecessarily simple assumptions about Silver's approach. The chances of it raining everyday this week are more complex than that just the numbers from each individual day and should also include:

How big of an area does the prediction involve?
Historical data for a given season.
And so forth.

For Silver there's also the issue that weather patterns influence each other. One day's likelihood of rain will impact the next day's likelihood. That's not the case for Colorado's likelihood of going Obama vs. Florida's likelihood. They're essentially independent of each other and thus probabilities of predicting one and not the other isn't particularly relevant.

If anything this person's critique of Silver doesn't comprehend the issues sufficiently enough to give a critique.

Also all of the bookies essential agreed with Silver. Silver was right. And the bookies are generally right. Silver just capitalized on that for marketing himself using essentially the same methods. He was basically just the right person in the right place a the right time to demonstrate how pundits make fools of themselves.

[Michael W. Kruse]

Whether the weather analogy is imperfect or not, the math is that Silver had a 6% chance of predicting the result he did. The RCP map shows the same result except for Florida, which was as close to a straight coin toss as it could be without being one at 50.3% by Sliver's data. Silver beat RCP because of a coin flip.

The most plausible explanation is that the predicted winner's chances were much higher in each of the swing states (90% and above?) than Silver's model estimated.

I'm not ragging on Silver. I think his work is interesting but the math doesn't add up. I think crowd-sourcing with the polls is probably as good or better than an individual experts models. I need some math to sway me and the professional math folks I'm reading aren't persuaded (and it has nothing to do with partisanship.)

[Dan]

Maybe, just maybe, and I'm just spitballing here now, perhaps he's trying to find ways to account for impacts on how regular national polling would affect turnout? After all, with poll data themselves becoming the news in itself, campaigns seem to use either favorable poll results to whip up enthusiasm or unfavorable poll results to increase the urgency.