Earlier in the day, I pointed out the string of bad polling over the last six months or more that has managed to miss right-leaning victories in Western elections. Others have begun to notice, too, especially in the UK after David Cameron’s resounding victory, preceded by predictions of doom from the media. For instance, the British Polling Council wants to conduct an investigation into how the UK pollsters got the call so badly wrong:
— BBC Breaking News (@BBCBreaking) May 8, 2015
David Axelrod, who worked on behalf of Labour (while Jim Messina worked for the Tories), also thinks that UK pollsters have some ‘splaining to do:
In all my years as journalist & strategist, I've never seen as stark a failure of polling as in UK. Huge project ahead to unravel that.
— David Axelrod (@davidaxelrod) May 8, 2015
It’s not just a problem in the UK, though. US pollsters and media organizations have blown calls here too, and poll aggregator Nate Silver says that it’s time to “worry about the state of the polling industry”:
- The final polls showed a close result in the Scottish independence referendum, with the “no” side projected to win by just 2 to 3 percentage points. In fact, “no” won by almost 11 percentage points.
- Although polls correctly implied that Republicans were favored to win the Senate in the 2014 U.S. midterms, they nevertheless significantlyunderestimated the GOP’s performance. Republicans’ margins over Democrats were about 4 points better than the polls in the average Senate race.
- Pre-election polls badly underestimated Likud’s performance in the Israeli legislative elections earlier this year, projecting the party to about 22 seats in the Knesset when it in fact won 30. (Exit polls on election night weren’t very good either.) …
[I]n the U.S., some pollsters have been caught withholding results when they differ from other surveys, “herding” toward a false consensus about a race instead of behaving independently. There may be more difficult times ahead for the polling industry.
Later in the evening, Silver added:
Our pre-election forecast, put together in conjunction with electionforecast.co.uk, did project a narrow Conservative plurality. And it was about in line with or slightly more bullish on the Conservatives than other forecasts and betting markets.
However, that’s not much of an excuse. The forecast assigned too little of a chance to an outcome like this one, especially given that there have been significant polling errors in the U.K. before. It’s a good lesson as we begin to plan our coverage for the 2016 U.S. election.
Politico’s Dylan Byers wondered whether Silver had talked himself out of his niche:
This is quite a notable statement. The former New York Times statistician gained national fame for correctly anticipating the outcome of the 2008, 2010 and 2012 U.S. elections. He did this largely by understanding how to read the polls, and by knowing which polls were worth reading. …
If Silver is declaring that the world has a polling problem, and that there may be more difficult times ahead for the polling industry, what is Silver’s added value in an election cycle? His ability to forecast elections is largely dependent on the accuracy of polling. Without that, what is his raison d’etre — other than to point out how bad polling caused him to make inaccurate forecasts?
It sounds more like Silver wants to preserve his niche. Bad polling and “herding” make his job much more difficult, especially the latter. Speaking out for reform may at first make it difficult to explain his analyses and projections without demonstrating some significant nod to the uncertainty in the polling industry, but the polls for 2016 now are nearly meaningless anyway, even if they’re hyper-accurate. The polling industry won’t get any better if those who rely on it just keep their mouths shut about its problems, as the string of examples Silver uses demonstrates.
Consumers of news and especially the analysts have to challenge pollsters to improve their batting average, and sooner rather than later. Otherwise, Silver and others like him will have nothing to add to Election Nights except to continue to recap the industry’s failures — as necessary as that may be.