Tarnished Silver: Reasons to doubt Nate’s model

Silver has said a lot about the model’s theoretical underpinnings, and what he has said is all ostensibly convincing. The polling numbers he uses as inputs are available for scrutiny, if (but only if) you’re on his list of pollsters. The weights he assigns to various polling firms, and the generating model for those weights, are public. But that still leaves most of the model somewhat obscure, and without a long series of tests—i.e., U.S. elections—we don’t really know that Nate is not pulling the numbers out of the mathematical equivalent of a goat’s bum.

Unfortunately, the most useful practical tests must necessarily come by means of structurally unusual presidential elections. The one scheduled for Tuesday won’t tell us much, since Silver gives both major-party candidates a reasonable chance of victory and there is no Ross Perot-type third-party gunslinger or other foreseeable anomaly to put desirable stress on his model. Silver defended his probabilistic estimate of the horserace this week by pointing out that other estimates, some based on simpler models and some based on betting markets, largely agree with his.

This is true, and it leaves us with only the question of what information Silver’s model may actually be adding to the field of alternatives. The answer could conceivably be “Less than none”, if his model (or his style of model-building) is inherently prone to getting the easy calls right and blowing up completely in the face of more difficult ones. (Taraji P. Henson Alert!) It is worth pointing out that a couple of statisticians have given us a potential presidential equivalent of the Marcels—a super-simple model that nailed the electoral vote the last two times (and that actually is fully specified).