Are we too sensitive to partisan skew in polls?

posted at 9:21 am on September 25, 2012 by Ed Morrissey

Critics of media polling have long complained about the sample skew that tends to favor Democrats in their surveys.  We do a lot of poll analysis here at Hot Air and we routinely compare the modeling to exit polling in past elections.  National Journal reports today that pollsters, especially those who work with media outlets, have begun to fight back against the criticism leveled by conservatives of institutional sample skew, claiming that they’re seeing trends on the ground that previous electoral models won’t capture:

Critics allege that pollsters are interviewing too many Democrats — and too few Republicans or independents — and artificially inflating the Democratic candidates’ performance. Pollsters counter that the results they are finding reflect slight changes in public sentiment — and, moreover, adjusting their polls to match arbitrary party-identification targets would be unscientific.

Unlike race, gender or age, all demographic traits for which pollsters weight their samples, party identification is considered an attitude that pollsters say they should be measuring. When party identification numbers change, it’s an indication of deeper political change that a poll can spot.

“If a pollster weights by party ID, they are substituting their own judgment as to what the electorate is going to look like. It’s not scientific,” said Doug Schwartz, the director of the Quinnipiac University Polling Institute, which doesn’t weight its surveys by party identification. …

On Monday, the news website Buzzfeed interviewed a Virginia-based blogger who re-weights public polls to reflect the partisan trends reported by automated pollster Rasmussen Reports. Dean Chambers, the blogger, then presents the adjusted data in charts on his website, unskewedpolls.com.

As of late Monday, Chambers’ website claimed that an average of polls conducted since Labor Day show Mitt Romney leading Obama, 52 percent to 44 percent. The website and its findings were trumpeted on the Drudge Report, the conservative-leaning news-aggregation site that has tended to highlight polls more favorable to Romney and less favorable to the president.

The pollsters claim that they’re seeing a big shift towards identification with the Democratic Party.  If so, then Gallup and Rasmussen have both missed it.  Both organizations routinely do general-population polling for partisan identification.  In fact, the latest state-by-state polling from Gallup (August 2012) shows that the shift has gone the other way:

Thus far in 2012, the two major parties have been closely matched nationally in terms of the absolute number of states each can claim as politically favorable, representing a dramatic change from 2008 and 2009 when the Democratic Party had an overwhelming advantage on this score. This doesn’t translate directly into likely election outcomes, given differences that can exist between the party leanings of adults versus registered voters, as well as differing turnout patterns and voting behavior of Republicans vs. Democrats in some states.

Polls are intended to be predictive.  In order to be predictive, the sample has to hew closely to the turnout model of the actual election. The best way to calculate that is to check the trends from the most recent election cycles.  One can get surprised by this when turnout shifts dramatically, as it did in 2008 — but that was in favor of the Democrats for a D+7 result, and it’s unlikely to happen a second time, especially after the all-even turnout model from the 2010 midterms.  That means that D+11 on national samples aren’t going to be terribly predictive of the outcome in November, nor would R+11 samples, and so it’s difficult to take those results seriously.  Furthermore, with just a few weeks before the election, pollsters need to start finding likely voters rather than just registered voters or general-population samples if they expect consumers to rely on them for predictions of voter behavior — again, the entire point of polling.

If there is one valid criticism of conservative poll analysis, it’s that we tend to focus on just party ID rather than a broader range of demographic categories — gender, age, income, and geography.  Most of that data exists in exit polling, too, so it isn’t terribly difficult to check, but it is time consuming.  However, if a national poll features a turnout model of D+11 or R+11, that’s enough to make the results unreliable without checking the rest of the demos, just as one with 60% men would be.

The pollsters complaining in this piece sound as though they resent the idea of having their models put into question at all.  They want consumers to simply swallow what they deliver without asking any questions.  If so, they’re relying on an outdated media-consumer model.


Related Posts:

Breaking on Hot Air

Blowback

Note from Hot Air management: This section is for comments from Hot Air's community of registered readers. Please don't assume that Hot Air management agrees with or otherwise endorses any particular comment just because we let it stand. A reminder: Anyone who fails to comply with our terms of use may lose their posting privilege.

Trackbacks/Pings

Trackback URL

Comments

Comment pages: 1 2 3

So if Romney wins, do all those who stay home share in the victory?

ddrintn on September 25, 2012 at 10:28 AM

I can’t speak for anyone else here, but for me, the Paulians are so completely irrational and irritating at a time of such dire consequences that if Obama wins, I’ll be suppressing an urge to rearrange the faces of any that I encounter.

I do not like Mitt Romney. However, elections in a constitutional republic like ours are not guaranteed to produce candidates that everybody likes. They’re designed — by choice, this is actually deliberate — to produce candidates that are compromises between sub-groups on a given side. We should expect to have to vote for compromise candidates.

I personally detested John McCain, but I voted for him. That does not make me a sellout, that just means that of the major candidates who had a chance to win, I would rather have had McCain in the White House than Obama. It’s a simple choice: which does one prefer? It does not signal some failure of the system if one does not particularly like either candidate.

I have no respect for Ron Paul; he’s a self-exalting, prating fool, and is provably wrong on every, single constitutional issue on which he weighs in. The man does not know the Constitution any better than an average 9th grader, and his lectures on the subject are an irritation and an embarrassment. And I am developing genuine animosity toward those effing imbeciles who are too effing infantile to vote for anybody else.

No, ddr, those who stay home don’t share in the victory, and if I have anything to do with it, you’ll get your @sses kicked in the case of a defeat.

philwynk on September 25, 2012 at 2:21 PM

Here’s the thing: polls before the last two weeks are liberated from reality. Their only constraints are not to appear of being too great an outlier. But the real test of who got it right won’t happen till election day.

So they spin, massage, fiddle with the data.

Romney will no doubt win by 7 or 8 points. The polls will all start to finally admit this come late October. And all the gains they made in creating an Obama winning narrative will be undone in what now looks like a reverse momentum for Romney.

To repeat: Romney by 7+. Go to the bank on it.

MaxMBJ on September 25, 2012 at 2:51 PM

Furthermore, with just a few weeks before the election, pollsters need to start finding likely voters rather than just registered voters or general-population samples if they expect consumers to rely on them for predictions of voter behavior — again, the entire point of polling.

No one seriously believes that, left or right. The entire point of polling is to supress voter turn out on the right.

That is the entire point.

And these polls do claim they are “likely voter” polls… Their definition of likely voters is simply “will they vote”. They are not really saying this is this predictive of turn out, but that is what the big difference is this year. They are not even trying.

I am most concerned that the Rasmussen ID poll will show a Democrat surge in September… that is still not going to tell you all that much about October, and even less about November, but with the bump from the convention there might be an increase in voter’s identifying as Dem.

I guess we will see.

petunia on September 25, 2012 at 3:43 PM

phlwynk… yes… Ron Paul people have the added luxury of never having to make anything he says, work in the real world… they are cowards who can’t bring themselves to support anyone who might actually have a chance… they leave their perch of constant complaint… then they would have nothing to talk about.

Ron Paul supporters were ready to defeat our Constiutional Republic by elevating their own ideology above the voice of a free thinking people. Those poor people in Nevada. They have no voice after the Paul people got done with their coup d’etat.

They are exactly the same as the Democrats… both are a small group of elitests who think they know enough to tell everyone else what to do, regardless of what people want to do. Despots of a different stripe are still despots.

petunia on September 25, 2012 at 3:50 PM

Polls are intended to be predictive.

Au contraire! Today, polls are intended to influence, not reflect, public opinion.

LASue on September 25, 2012 at 4:26 PM

I just do not trust those polls anymore. I don’t trust much of anything as far as that is concerned.

Terrye on September 25, 2012 at 4:56 PM

No, a supremely butthurt Palinista (disclaimer: I am a Palinista too) who never got over it and decided to take it out on everyone else.

kim roy on September 25, 2012 at 2:18 PM

Bull. Shit. That is ALWAYS the red herring ‘bots new and old throw at me and others when asked some tough questions. We either get some reference to Palin — as if support for Palin is entirely incongruous with being skeptical of yet another GOPe mannequin squish — or we get some idiotic ad hominem such as that from War Planner, xblade, wargamer666 or something (don’tcha just love those studly warlike monikers?) and others here.

ANSWER MY DAMN QUESTIONS.

I just do not trust those polls anymore. I don’t trust much of anything as far as that is concerned.

Terrye on September 25, 2012 at 4:56 PM

But yet you trusted them just fine back when they were showing Mitt to be some Superelectable Invincible Superman.

ddrintn on September 25, 2012 at 5:58 PM

Au contraire! Today, polls are intended to influence, not reflect, public opinion.

LASue on September 25, 2012 at 4:26 PM

Which (tada!!!) is why we have Mitt Romney, as stated above.

ddrintn on September 25, 2012 at 6:00 PM

ddrintn… au contraire!

We have Mitt because he was the best and beat all the others to a pulp.

Who pray tell got more votes than Mitt? Certainly not anyone who actually had their name on the ballot!

You see in this country we have a funny system… the guy who gets the most votes… is the nominee of the party. And belly aching every election because you don’t like that person adds nothing to your spikey personality.

I did not see Palin’s name on any ballots… and yet you are bitter she did not win. Prehaps you expected her to magically appear with a crown?

You still back a person who didn’t even have the guts to run!!!! Please tell us how that makes you the smart one?

petunia on September 25, 2012 at 6:15 PM

Unlike the many Romney lovers on this site, I am not in denial with the polls. Even Rasmussen has Obama winning if at a closer margin and he can be accused of over sampling Republicans on a presidential Election year.

Presidential Elections are about demographics and gender mostly, not party affiliation. People turn out to vote on Presidential Election years unlike midterm Election years where the minority Party fights to be relevant in Congress. Democrats have edged Republicans in percentage in Presidential Election years for MANY MANY MANY years now regardless of how well Republicans did in some of the midterm Elections….because of the MINORITY vote that always turns out during presidential Election years.
Blacks, Hispanics, Asians, Arabs, Indians and all naturalized non white immigrants turn out in BIG numbers to vote Democrats during Presidential Election years. Democrats have done a great job convincing minorities that Republicans are white supremacists that would take away their rights if they control the presidency. Women also turn out in bigger numbers during presidential Election years and it’s advantage Democrats…. This is why Democrats always edge Republican in Presidential Election year party affiliation.

Because of the huge minority deficit (a percentage that keeps growing), Republicans have to win over 60% of whites to make it close. Are there enough whites out there that will turn out for Romney to give him 60+ % of the white vote and will the white vote be huge enough to offset the minority and liberal white deficit? That’s where the focus should be, not party affiliation which does not matter much during Presidential Election years.

The minority vote will turn out big for Obama and they could care less about the economy or foreign policies (if anything the Arab vote will help Obama in states like Michigan or Wisconsin). Will white conservatives be energized enough behind a candidate they don’t really like? A true conservative would definitely get enough white conservatives and religious voters out there…Not sure Mitt Romney will. Republicans nominated the wrong candidate (as usual), period…

jules on September 25, 2012 at 10:34 PM

ddrintn… au contraire!

We have Mitt because he was the best and beat all the others to a pulp.

LOL, to a pulp? The guy only started getting more than 25% of the vote by way of attrition. Anyway, that’s a circularity of sorts, not unexpected when the past year has been the Mittbot Logical Fallacy Festival. “Mitt won the nomination.”
“Yeah, because he was pimped for three years by ‘bots waving the sacred polls around showing he was The Only One Who Can Beat Obama, and enough sheeple swallowed the dreck.” “Au contraire! He won because he got the most votes!” ROFL

Who pray tell got more votes than Mitt?

NotRomney, for the most part.

I did not see Palin’s name on any ballots…

petunia on September 25, 2012 at 6:15 PM

You didn’t see Palin’s name in any of my original comments, either. In fact, Palin’s name is usually brought up by ‘bots grasping for a red herring to try to draw attention away from the weakness of their Squish Messiah.

ddrintn on September 26, 2012 at 12:50 AM

They are exactly the same as the Democrats… both are a small group of elitests who think they know enough to tell everyone else what to do, regardless of what people want to do. Despots of a different stripe are still despots.

petunia on September 25, 2012 at 3:50 PM

Exquisite irony coming from a ‘bot. LOL

This is why Democrats always edge Republican in Presidential Election year party affiliation.

Because of the huge minority deficit (a percentage that keeps growing), Republicans have to win over 60% of whites to make it close. Are there enough whites out there that will turn out for Romney to give him 60+ % of the white vote and will the white vote be huge enough to offset the minority and liberal white deficit? That’s where the focus should be, not party affiliation which does not matter much during Presidential Election years.

The minority vote will turn out big for Obama and they could care less about the economy or foreign policies (if anything the Arab vote will help Obama in states like Michigan or Wisconsin). Will white conservatives be energized enough behind a candidate they don’t really like? A true conservative would definitely get enough white conservatives and religious voters out there…Not sure Mitt Romney will. Republicans nominated the wrong candidate (as usual), period…

jules on September 25, 2012 at 10:34 PM

Well, turnout in 2004 was about equal; but that was GOP turning out in support of an incumbent Republican president who was being challenged by a Dem nominee unliked for the most part by the Dem base. And Bush still won only by the skin of his teeth.

Anyway, your overall point is valid. This ridiculous fantasy that turnout in 2012 is going to mirror that of 2010 is just that, a ridiculous fantasy.

ddrintn on September 26, 2012 at 1:00 AM

This ridiculous fantasy that turnout in 2012 is going to mirror that of 2010 is just that, a ridiculous fantasy.

ddrintn on September 26, 2012 at 1:00 AM

Thank you for proving that if someone speaks long enough they will eventually get something right.

I don’t know what the 2012 electorate is going to look like but anyone who expects something close to 2010 is both ignorant of history and probably wrong.

alchemist19 on September 26, 2012 at 3:36 AM

Republicans may do a little better in 2012 than in 2010. Reason is math. The difference between Democrats and Republicans registered has narrowed since 2010. So far in 2012, Democrat registrations are running at only 10% of the rate of Dem registrations in 2008. Republican registrations are 200% of 2008 and Independents are 400%. Democrats just don’t have the voter advantage they had. Rasmussen says party ID is now R+4 where it was D+7 in 2008.

New polls out tonight from Quinnipiac University/CBS News/New York Times. Ohio D+9 sample. Florida D+9 sample. Pennsylvania a D+11 sample. All three polls absolutely ridiculous samples.

http://www.cbsnews.com/8301-250_162-57520341/poll-obama-opens-substantial-leads-in-key-swing-states/

Crosstabs are on page 3.

Total fantasy polling.

crosspatch on September 26, 2012 at 4:45 AM

With all the polls out there it,s sometimes hard to find the real truth.For example Rasmussen polls are considered one of the best out there.in his latest polls he finds the generic break down Reps 44% Dems 43%.The Dems have never been above 43% for the last year yet even Rasmussen weights the Dems +2 or +3 in all of his polls.So when you see one of his polls showing Obama 48% to Romney 47% is that the real number?Also he is over weighting the young vote because most of them use Cell phones instead of land lines.I guess this is why he has such a large margin of error + or – 4%I think it comes down to 3 states Fla.,Ohio and Virgina .Romney wins these and he wins the Pres.

logman1 on September 26, 2012 at 8:00 AM

Thank you for proving that if someone speaks long enough they will eventually get something right.

alchemist19 on September 26, 2012 at 3:36 AM

You nor any other ‘bot has ever proven me wrong before. Sitting there calling me a poo poo head in itself just doesn’t cut it.

For example Rasmussen polls are considered one of the best out there.in his latest polls he finds the generic break down Reps 44% Dems 43%.The Dems have never been above 43% for the last year yet even Rasmussen weights the Dems +2 or +3 in all of his polls.

logman1 on September 26, 2012 at 8:00 AM

When was the last presidential election in which Reps outnumbered Dems?

ddrintn on September 26, 2012 at 9:33 AM

Well, turnout in 2004 was about equal; but that was GOP turning out in support of an incumbent Republican president who was being challenged by a Dem nominee unliked for the most part by the Dem base. And Bush still won only by the skin of his teeth.

Exactly. 2004 was in the middle of wars in Afganistan and Iraq, Karl Rove bused in the religious right from every churhes in the country and Democrats were still +1. Bush had to get a big junk of the latino vote to barely make it.
Again, Presidential Elections are about demographics and gender, not party affiliation. All Romney has is white men and they may not be enough to offset the minority and women vote in a Presidential year where people turn out (unlike midterm election years).

jules on September 26, 2012 at 9:40 AM

STUPID CONSERVATIVES! HOW DARE THEY QUESTION US?

Yeah, I find this totally convincing.

tom on September 26, 2012 at 1:31 PM

When was the last presidential election in which Reps outnumbered Dems?

ddrintn on September 26, 2012 at 9:33 AM

1. You talking about voting in elections or overall count?
2. Are you assuming that all Democrats will vote for the liberal view?

Here’s something to consider:
Harris Poll: Self-Identified Conservatives Now Outnumber Liberals 2-to-1 – Democrats hit new low
Harris Poll press release ^ | January 30, 2003
http://www.freerepublic.com/focus/news/832859/posts

avagreen on September 26, 2012 at 5:40 PM

1. You talking about voting in elections or overall count?
2. Are you assuming that all Democrats will vote for the liberal view?

avagreen on September 26, 2012 at 5:40 PM

I’m talking about what I said. When was the last time that Reps outnumbered Dems in a presidential election?

ddrintn on September 27, 2012 at 9:39 AM

Comment pages: 1 2 3