Are we too sensitive to partisan skew in polls?

Critics of media polling have long complained about the sample skew that tends to favor Democrats in their surveys.  We do a lot of poll analysis here at Hot Air and we routinely compare the modeling to exit polling in past elections.  National Journal reports today that pollsters, especially those who work with media outlets, have begun to fight back against the criticism leveled by conservatives of institutional sample skew, claiming that they’re seeing trends on the ground that previous electoral models won’t capture:

Critics allege that pollsters are interviewing too many Democrats — and too few Republicans or independents — and artificially inflating the Democratic candidates’ performance. Pollsters counter that the results they are finding reflect slight changes in public sentiment — and, moreover, adjusting their polls to match arbitrary party-identification targets would be unscientific.

Unlike race, gender or age, all demographic traits for which pollsters weight their samples, party identification is considered an attitude that pollsters say they should be measuring. When party identification numbers change, it’s an indication of deeper political change that a poll can spot.

“If a pollster weights by party ID, they are substituting their own judgment as to what the electorate is going to look like. It’s not scientific,” said Doug Schwartz, the director of the Quinnipiac University Polling Institute, which doesn’t weight its surveys by party identification. …

On Monday, the news website Buzzfeed interviewed a Virginia-based blogger who re-weights public polls to reflect the partisan trends reported by automated pollster Rasmussen Reports. Dean Chambers, the blogger, then presents the adjusted data in charts on his website, unskewedpolls.com.

As of late Monday, Chambers’ website claimed that an average of polls conducted since Labor Day show Mitt Romney leading Obama, 52 percent to 44 percent. The website and its findings were trumpeted on the Drudge Report, the conservative-leaning news-aggregation site that has tended to highlight polls more favorable to Romney and less favorable to the president.

The pollsters claim that they’re seeing a big shift towards identification with the Democratic Party.  If so, then Gallup and Rasmussen have both missed it.  Both organizations routinely do general-population polling for partisan identification.  In fact, the latest state-by-state polling from Gallup (August 2012) shows that the shift has gone the other way:

Thus far in 2012, the two major parties have been closely matched nationally in terms of the absolute number of states each can claim as politically favorable, representing a dramatic change from 2008 and 2009 when the Democratic Party had an overwhelming advantage on this score. This doesn’t translate directly into likely election outcomes, given differences that can exist between the party leanings of adults versus registered voters, as well as differing turnout patterns and voting behavior of Republicans vs. Democrats in some states.

Polls are intended to be predictive.  In order to be predictive, the sample has to hew closely to the turnout model of the actual election. The best way to calculate that is to check the trends from the most recent election cycles.  One can get surprised by this when turnout shifts dramatically, as it did in 2008 — but that was in favor of the Democrats for a D+7 result, and it’s unlikely to happen a second time, especially after the all-even turnout model from the 2010 midterms.  That means that D+11 on national samples aren’t going to be terribly predictive of the outcome in November, nor would R+11 samples, and so it’s difficult to take those results seriously.  Furthermore, with just a few weeks before the election, pollsters need to start finding likely voters rather than just registered voters or general-population samples if they expect consumers to rely on them for predictions of voter behavior — again, the entire point of polling.

If there is one valid criticism of conservative poll analysis, it’s that we tend to focus on just party ID rather than a broader range of demographic categories — gender, age, income, and geography.  Most of that data exists in exit polling, too, so it isn’t terribly difficult to check, but it is time consuming.  However, if a national poll features a turnout model of D+11 or R+11, that’s enough to make the results unreliable without checking the rest of the demos, just as one with 60% men would be.

The pollsters complaining in this piece sound as though they resent the idea of having their models put into question at all.  They want consumers to simply swallow what they deliver without asking any questions.  If so, they’re relying on an outdated media-consumer model.

• Gallup: Abortion views steady as media ignores Gosnell trial
• Suffolk poll puts Markey up 17 over Gomez
• PPP: South Carolina a dead heat

Blowback