The Columbia Journalism Review isn’t known for its stirring defenses of conservatives, so their analysis of the Harris poll that suggested that 45% of Republicans are Birthers and 24% believe Obama to be the Antichrist is remarkable for its conclusion.  Greg Marx pegs his analysis on a deconstruction by ABC News polling director Gary Langer and wonders why the national media was so credulous in reporting a poll obviously aimed to help author John Avlon sell his book:

But in order both to be credible and to be perceived as credible, that undertaking has to come from a place of open inquiry, not from an expectation that the results will boost book sales, create a news cycle, or confirm a political narrative. The whole appeal of polling is its promise (sometimes oversold) to produce objective, almost scientific data that tells us something about the state of our politics. When that enterprise itself becomes an exercise in political gamesmanship, it may foster cynicism about the utility and quality of any effort to collect information. If any given piece of news is valuable only insofar as it advances a particular political view, then they are all in a sense equivalent, which is to say worthless.

That’s not to say that political perspectives don’t have a place in journalism (they do) or that our political views won’t shape the way we understand information (they will). But we still need to protect a place for facts and evidence in our political debate, and to do that we need to push back against rhetorical opportunism and statistical sloppiness on all sides. Journalists can do this by not cutting corners when reporting and compiling data—but also by making the case for why methodological rigor is valuable, and by providing readers with the tools to evaluate information themselves.

This, in the end, is the real value in Langer’s post—it explains why the Harris poll is flawed in a way that will hopefully prompt closer scrutiny of storyline it perpetuated. While this type of direct engagement between media outlets is somewhat unusual, Langer has a clear incentive here: as a pollster, he needs to uphold the integrity, credibility, and rigor of his profession. It’s a lesson for other members of the press to keep in mind.

Langer ripped Harris for its methodology, starting with its interactive model based on incentivizing through cash and gifts, but that was just to start:

The problems are fundamental. “Some people have said” is a biasing introductory phrase; it imbues the subsequent statements with an air of credibility – particularly when you don’t note that others say something else. (That approach can have problems of its own; the “some people” vs. “other people” format implies equivalence.)

The subsequent statements, for their part, are classically unbalanced – there’s no alternative proposition to consider. A wealth of academic literature, neatly summarized here, demonstrates that questions constructed in this fashion – true/false, agree/disagree – carry a heavy dose of what’s known as acquiescence bias. They overstate agreement with whatever’s been posited, often by a very substantial margin. (This reflects avoidance of cognitive burden, which tends to happen disproportionately with less-educated respondents, as is reflected in Harris’ results.)

Using all negative statements, rather than a mix of negative and positive ones, reflects another non-standard approach, one that can further bias responses. (The ordering of  items, unclear in the Harris release, can be troublesome as well.)

Another problem, which I discuss here, is the challenge of over-literalism in evaluating survey results of this type. Rather than answering disparaging poll questions literally, people who are ill-disposed toward the subject may simply use these questions as an opportunity to express their general antipathy – not as a thought-out endorsement of the specific posit. And the use of hot-button invective is ill-advised in its own right; respondents may just blow it back.

Langer also compares the result to more traditional polls taken by Pew during and just after the election by linking to an observation by Time’s Michael Scherer.  Instead of 57% of Republicans believing that Obama is a secret Muslim, Pew found only 17% of Republicans believing that, clearly a fringe view in the GOP even when those sentiments burned most hotly.

The big question here isn’t that Harris did a push poll, or that Avlon exploited it to sell his books.  Push polls have been around for a very long time.  CJR’s point is that the media apparently fell for it, spreading misinformation, rather than doing some routine fact-checking to determine the poll’s credibility.  I’d say that indicates another age-old trend of accepting data points without confirmation when they fit an accepted worldview — and in the national media, a poll declaring Republicans to mainly be lunatics would certainly fit that explanation.

Update: Kathy Shaidle has more thoughts.