Premium

Confirmed: Pollsters stunk at record levels in 2020 -- but why?

AP Photo/Eric Gay

It’s official: Presidential polls had their worst aggregate performance in decades. That’s not just the opinion of pundits — it’s also the opinion of pollsters. A new report from an industry group concluded that the amount of errors and their “unusual magnitude” combined for a miss the likes of which hadn’t been seen in decades.

And it wasn’t just a Trump factor at play, either, the Washington Post concedes:

Public opinion polls in the 2020 presidential election suffered from errors of “unusual magnitude,” the highest in 40 years for surveys estimating the national popular vote and in at least 20 years for state-level polls, according to a study conducted by the American Association for Public Opinion Research (AAPOR).

The AAPOR task force examined 2,858 polls, including 529 national presidential race polls and 1,572 state-level presidential polls. They found that the surveys overstated the margin between President Biden and former president Donald Trump by 3.9 points in the national popular vote and 4.3 percentage points in state polls.

Polls understated the support for Trump in nearly every state and by an average of 3.3 percentage points overall. Polls in Senate and gubernatorial races suffered from the same problem.

“There was a systematic error that was found in terms of the overstatement for Democratic support across the board,” said Josh Clinton, a Vanderbilt University political science professor who chaired the 19-member task force. “It didn’t matter what type of poll you were doing, whether you’re interviewing by phone or Internet or whatever. And it didn’t matter what type of race, whether President Trump was on the ballot or was not on the ballot.”

In fact, the biggest failure didn’t come in the presidential race. It came in the House and Senate polling that had pointed to a Democratic sweep:

Polling in senatorial and gubernatorial races showed a similar pattern, overstating the margin for Democratic candidates versus their Republican opponents. When state-level presidential polls were removed from the sample, the error level was even higher. For example, polling pointed to possible Democratic gains in House races. Instead, Republicans gained seats.

That actually helps eliminate a few possible explanations. Two immediate theories about the polling failure revolved around responsiveness of Trump voters. One theory had them as too shy to share their support of Trump with pollsters, while another one had Trump voters maliciously misleading in their survey answers to discredit pollsters. While the variance on presidential polling to outcomes was large, the variance on House and Senate races was even larger — which means that the problem wasn’t the respondents.

So what was the problem? AAPOR essentially threw up its hands on that question:

But unlike 2016, when pollsters could pinpoint factors like the education divide as reasons they underestimated Donald Trump and offer specific recommendations to fix the problem, the authors of the new American Association for Public Opinion Research report couldn’t put their finger on the exact problem they face now. Instead, they’ve stuck to rejecting the idea that they made the same mistakes as before, while pointing to possible new reasons for inaccuracy. …

Decreasing response rates have been a major source of concern for pollsters for more than a decade. But the politicization of polling during the Trump era — including the feedback loop from the former president, who has falsely decried poll results he doesn’t like as “fake” or deliberately aimed at suppressing enthusiasm for answering polls among GOP voters — appears to be skewing the results, with some segment of Republicans refusing to participate in surveys.

But pollsters say they can’t be sure that’s the main reason, because you never know exactly whom you’re not talking to. …

The most plausible — yet still unproven — theory is that the voters the polls are reaching are fundamentally different from those they are not. And Trump’s rantings about the polls being “fake” or rigged only exacerbate that problem.

Actually, the polling failures are what “exacerbated” the problem, and those existed long before Trump was on the scene. Media-based polling has been particularly suspect for a long time, including that performed by the Washington Post. Sample composition and weighting get manipulated almost to the very last poll of a cycle, when sudden shifts in both get glossed over as actual public-opinion shifts and not deliberate polling corrections.

Speaking of manipulation: Remember the atrocious polling cycle in 2014, for instance, when most major pollsters missed the big GOP gains in the Senate? Nate Silver and David Axelrod sent up warning flares about the polling industry in 2015, and not just about the US but in the UK and other Western nations. Neither the Post nor Politico even mention 2014’s poll performance, which was almost as bad as 2020’s. Why? Any mention of 2014 would negate the “Trump made it worse” explanation.

This isn’t a Trump problem. It’s a credibility issue that pollsters and media outlets have created for themselves, and people are refusing to cooperate as a direct result of those failures. Trump’s complaints might amplify that response somewhat, but that’s not to say that Trump started that credibility issue — or that he’s wrong about it, either. That’s also a point that the Post and Politico largely evade in their attempts to shift blame to Trump. In fact, as the AAPOR report makes painfully clear, Trump is actually correct about polling performance in the US.

So what will AAPOR do about the problem? Don’t expect it to get better in the short term, since they and their constituent polling orgs seem intent on evading the real problems. As long as they keep failing, people will refuse to cooperate. Perhaps AAPOR could suggest that pollsters stop conducting surveys for media outlets, which too often get used and manipulated to generate news rather than take the legitimate temperature of the electorate. However, since that’s where many of these orgs make most of their money, bet on the decline accelerating instead.