Selection bias at Politifact?

posted at 9:30 am on February 10, 2011 by Ed Morrissey

For quite a while, a debate has simmered about whether PolitiFact operates from a political bias.  The Pulitzer Prize-winning feature from the St. Petersburg Times in Florida rates the truthfulness of public statements by politicians and activists on a scale ranging from True to Pants On Fire.  Republicans have complained for quite a while that PolitiFact aims more at the GOP, especially when PolitiFact named the allegation that ObamaCare was a government takeover of health care the “Lie of the Year.”

Now, Eric Ostermeier at Smart Politics has published the results of a study he has made that shows Republicans getting much harsher treatment than Democrats over the last 13 months:

PolitiFact assigns “Pants on Fire” or “False” ratings to 39 percent of Republican statements compared to just 12percent of Democrats since January 2010 …

But although PolitiFact provides a blueprint as to how statements arerated, it does not detail how statements are selected.

For while there is no doubt members of both political parties make numerous factual as well as inaccurate statements – and everything in between – there remains a fundamental question of which statements (by which politicians) are targeted for analysis in the first place.

A Smart Politics content analysis of more than 500 PolitiFact stories from January 2010 through January 2011 finds that current and former Republican officeholders have been assigned substantially harsher grades by the news organization than their Democratic counterparts.

In total, 74 of the 98 statements by political figures judged “false” or “pants on fire” over the last 13 months were given to Republicans, or 76 percent, compared to just 22 statements for Democrats (22 percent).

Ostermeier notes that the breakdown of statements reviewed is more or less evenly split, with 50.4% of the statements from Republican public officials, 47.2% from Democrats, and the small remainder from independents.  But that’s curious in itself, as Ostermeier later points out:

What is particularly interesting about these findings is that the political party in control of the Presidency, the US Senate, and the US House during almost the entirety of the period under analysis was the Democrats, not the Republicans.

To remind everyone, that control wasn’t exactly an even split, either.  Democrats had sixty percent of the Senate seats, and close to the same percentage in the House.  A Democrat was in the White House.  Republicans controlled nothing in Washington.  What made Republicans so attractive to PolitiFact, especially if the oft-expressed purpose of the Fourth Estate is to hold government accountable?

Ostermeier notes that Politifact itself has expressed a rather ad hoc approach to selection:

When PolitiFact Editor Bill Adair was on C-SPAN’s Washington Journal in August of 2009, he explained how statements are picked:

We choose to check things we are curious about. If we look at something and we think that an elected official or talk show host is wrong, then we will fact-check it.”

If that is the methodology, then why is it that PolitiFact takes Republicans to the woodshed much more frequently than Democrats?

The answer to the overall question could still be that Republicans tell more Pants on Fire and False statements, and that Politifact is merely a disinterested referee.  However, the numbers suggest that PolitiFact is more “curious” about Republican statements and less curious about Democratic statements, even when Democrats vastly outnumbered Republicans in the halls of power.  And that certainly is … curious.


Related Posts:

Breaking on Hot Air

Blowback

Note from Hot Air management: This section is for comments from Hot Air's community of registered readers. Please don't assume that Hot Air management agrees with or otherwise endorses any particular comment just because we let it stand. A reminder: Anyone who fails to comply with our terms of use may lose their posting privilege.

Trackbacks/Pings

Trackback URL

Comments

Dan Rather call yer office.

Akzed on February 10, 2011 at 9:34 AM

Yeah, with the Democrats in complete control of the agenda, you’d think they would get a little more scrutiny. Especially since they lie so much.

forest on February 10, 2011 at 9:34 AM

Liars

Wade on February 10, 2011 at 9:36 AM

Politifact does have a point. I mean, afterall, how do you fact check statements about Guam tipping over and having to pass a bill so that you can find out what’s in it?

WordsMatter on February 10, 2011 at 9:36 AM

If Politifact was honest, fair and unbiased, it wouldn’t have won a Pulitzer.

cartooner on February 10, 2011 at 9:38 AM

The answer to the overall question could still be that Republicans tell more Pants on Fire and False statements, and that Politifact is merely a disinterested referee.

Yup, sure could…in a make believe, I’ll believe anything, “I always prefer to give the benefit of the doubt”…no matter what the facts are world..

winston on February 10, 2011 at 9:38 AM

They’ve been caught more than once, just making it up as they go along.

Death panels anyone?
Maybe just out and out lie about the Bush tax cuts expiring, and who would be effected.

It takes more to “fact check” than just some J-Tard slapping the words “fact check” on their latest leftist editorial.

MNHawk on February 10, 2011 at 9:39 AM

http://www.theblaze.com/stories/shock-graphics-show-severity-of-proposed-obama-budget-cuts/
A graph showing Obama’s proposed tax cuts. BEAUTIFUL!

mizflame98 on February 10, 2011 at 9:39 AM

Actually, statements such as “There are no rules here… we make them up as we go along,” are in fact, 100 percent correct.

WordsMatter on February 10, 2011 at 9:40 AM

To remind everyone, that control wasn’t exactly an even split, either. Democrats had sixty percent of the Senate seats, and close to the same percentage in the House. A Democrat was in the White House. Republicans controlled nothing in Washington. What made Republicans so attractive to PolitiFact, especially if the oft-expressed purpose of the Fourth Estate is to hold government accountable?

But it measured the veracity of statements made by current and former officeholders. Just because a politician isn’t in office doesn’t mean they don’t have significant influence in the media (see, e.g. the 57 Palin posts made yesterday) and should be held accountable.

The ad hoc method of selection is more troublesome, but it’s also not out of the realm of possibility that Republicans have been making more outrageous statements over the past couple years – the party that’s out of power tends to do that (e.g. “Bush’s illegal war”, “close down Gitmo”, etc.)

Proud Rino on February 10, 2011 at 9:42 AM

Maybe Politifact didn’t think phrases such as the “Good and Welfare Clause” weren’t curious enough.

WordsMatter on February 10, 2011 at 9:43 AM

Credibility is a b*tch.

Alinsky on February 10, 2011 at 9:43 AM

An undiscovered lie is not a lie.

BobMbx on February 10, 2011 at 9:43 AM

Who’s watching the Watchmen?

SKYFOX on February 10, 2011 at 9:43 AM

http://politifact.com/truth-o-meter/statements/2010/aug/04/sarah-palin/sarah-palin-said-democrats-planning-largest-tax/

President Barack Obama’s plan would increase income taxes for individuals making more than $200,000 and for couples making more than $250,000, with indexing for inflation.

There was never a “plan.” This is when Politilie left no doubt they were utter frauds.

MNHawk on February 10, 2011 at 9:44 AM

I think the most stunning example of Politifact’s lapdoggery is the “pant on fire” rating they gave to Glenn Back when he stated, quite accurately, that John Holdren once advocated involuntary sterilization and other draconian measures to curb population.

It was all there in black and white and yet somehow Politifact managed to ignore it and but Holdrens defense hook line and sinker.

DJ Rick on February 10, 2011 at 9:46 AM

As a CNN exec once told me in confidence: “My job isn’t to tell the news. My job is to decide what the news is, and convince you of that.”

Vashta.Nerada on February 10, 2011 at 9:46 AM

Why would you check a statement that “everybody knows” is true?

“The trouble with our liberal friends is not that they are ignorant, but that they know so much that isn’t so.” R. Reagan, 1964

ss396 on February 10, 2011 at 9:46 AM

So according to Politifact, when Joe Biden told a reporter that the government would control insurance companies through the implemntation of Obamacare, he was lying… yeah.

WordsMatter on February 10, 2011 at 9:49 AM

Hey Politifact, fact check this: “As sure as any law of physics, as government expands, liberty contracts.” – Ronald Reagan

WordsMatter on February 10, 2011 at 9:50 AM

Yeah. Notice also that we had a GOP congressman resign yesterday over some kind of scandal (it has to be more than a shirtless picture). And the lefties are all aflutter about the rumors of a Boehner scandal soon to come from the NYT and/or National Enquirer.

The MBM Destroy All Republicans machine is already in midseason form. I hate to think how bad it’ll be over the next 2 years.

tsj017 on February 10, 2011 at 9:50 AM

Politifact does have a point. I mean, afterall, how do you fact check statements about Guam tipping over and having to pass a bill so that you can find out what’s in it?

WordsMatter on February 10, 2011 at 9:36 AM

Unfortunately that part was true. No one including Pelosi had read the 2000+ pages to find out what Soros had put in it.

Kataklysmic on February 10, 2011 at 9:52 AM

I stinks to have to see all the daily garbage spewed by a complicit media.

HomeoftheBrave on February 10, 2011 at 9:55 AM

It is simply an institional bias. When a liberal hears a liberal talking point he/she accepts it as fact, there isn’t any reason to be curious about something you accept as fact.

When a different perspective is brought up, one that conflicts with what you have accepted, you go to the sources that you accept as credible to check it.

jpmn on February 10, 2011 at 10:00 AM

I live in St. Pete. The TIMES is our primary daily paper. It has long been known that they only print on the left hand pages. They are rarely hampered by the need to present the actual truth of anything. In point of fact, they recently hired Daniel Ruth, a self-described leftist columnist from the Tampa Tribune, after he had been fired by the Trib. The Trib attributed his dismissal as due to budgetary constraints, but it is generally agreed that he was too left, even for them. At the TIMES, he’s sort of a middle-of-the-road kind of guy. And people wonder why newspapers are in decline… Sheesh.

Crusader Rabbit on February 10, 2011 at 10:03 AM

Why would you check a statement that “everybody knows” is true?

“The trouble with our liberal friends is not that they are ignorant, but that they know so much that isn’t so.” R. Reagan, 1964

ss396 on February 10, 2011 at 9:46 AM

Yep, that’s the problem when the newsrooms are 90% leftist. They get their preconceived notions validated by each other over lunch and after work drinks, and the next thing you know, they are reporting fabricated quotes by Palin or Limbaugh based on somebody else’s reporting without looking for the primary source. Or Politifact doesn’t even bother to look into big whoppers by Obama or Pelosi because they are too busy digging into the opposition’s “government takeover” comments. But that’s just the “soft bias”, and it’s become apparent that there’s plenty of intentional and deliberate bias too.

forest on February 10, 2011 at 10:04 AM

“Selection bias at Politifact?”

Anyone who has followed Obamateurism Of The Day would know there is a bias against Republicans, and I am not referring to the gaffes, but the Pants on Fire lies cataloged.

Dusty on February 10, 2011 at 10:04 AM

That paper is part of the MSM, so I automatically assume a liberal bias.

Ward Cleaver on February 10, 2011 at 10:10 AM

PPP
Progressive Propagandized Pravda

esnap on February 10, 2011 at 10:12 AM

If we look at something and we think that an elected official or talk show host is wrong, then we will fact-check it.”

Hmmm…this strikes me as not just a bias but an inappropriate way of doing fact checking. They make an assumption… “we look at something and we think that an elected official or talk show host is wrong…” Then they proceed from that assumption. Shouldn’t they simply base their fact checking on statements they…”are curious about” without the qualifier?

Deanna on February 10, 2011 at 10:15 AM

Perhaps PolitiFact should be assigned a “Pants on Fire” rating.

GarandFan on February 10, 2011 at 10:20 AM

But it measured the veracity of statements made by current and former officeholders. Just because a politician isn’t in office doesn’t mean they don’t have significant influence in the media (see, e.g. the 57 Palin posts made yesterday) and should be held accountable.

The ad hoc method of selection is more troublesome, but it’s also not out of the realm of possibility that Republicans have been making more outrageous statements over the past couple years – the party that’s out of power tends to do that (e.g. “Bush’s illegal war”, “close down Gitmo”, etc.)

Proud Rino on February 10, 2011 at 9:42 AM

Red herring anyone? Keep flogging that chicken, I’m sure it makes you feel better.

fossten on February 10, 2011 at 10:22 AM

Most newspapers get their information from the same sources and are at the mercy of said sources for the truth. Local issues may be slanted to conform with the overall messages being received from outside sources. The mainstream media is a propaganda tool of the left and should be judged accordingly.

volsense on February 10, 2011 at 10:22 AM

the results of a study he has made that shows Republicans getting much harsher treatment than Democrats over the last 13 months:
============

Republicans tell’n da truth,with Progressive Whining
Backlash and Liberals Anxiety prior to and after the
Midterms!!!!!!!!!!!!

canopfor on February 10, 2011 at 10:24 AM

“We choose to check things we are curious about. If we look at something and we think that an elected official or talk show host is wrong, then we will fact-check it.”

Boy that’s a big mouse in your pocket Mr. Adair. One would have to wonder how Adair would handle a two-year flood of Ed’s Obamateurisms???

Rovin on February 10, 2011 at 10:26 AM

Since the St. Pete Times is both the main paper in my area and the sponsor of PolitiFact, anyone from around here knows PF is biased. The paper itself is a shameless liberal rag, with their biased views permiating every section of the paper — news, arts, sports, even the comics page tilts left.

Most conservatives around here have recognized what has finally been shown statistically — PF deliberately selects statements by conservatives/republicans it knows it can prove false, then runs with them. Hardly fair and balanced!

postaldog on February 10, 2011 at 10:31 AM

Agree with postaldog. St. Pete Times is far-left, so of course politifact would be as well. Poltifact is absolutely worthless.

WarEagle01 on February 10, 2011 at 10:34 AM

I never heard of Politifact before this story, so I’m NOT sorry:
- if their feelings are hurt that neither I nor MILLIONS OF AMERICANS have no idea who they are
- what they do
- how long they’ve done it
- and never noticed what little or marginal impact they have ever had on anything past, present or future…

Feel free to carve that in stone… THAT’S A FACT, JACK!

Khun Joe on February 10, 2011 at 10:47 AM

The New York Times has their own built in Poltifact: Paul Krugman, Frank Reich, and Mo Dowd. And of course the Times would tell you there not biased:

First of all, let’s put the New York Times in their partisan box. On November 1st, the Times Editorial Board put out this list of endorsements, (and no comments feature):

NEW YORK
Poll hours: 6 a.m. to 9 p.m.
UNITED STATES SENATE
Charles Schumer (D)
Kirsten Gillibrand (D)
HOUSE OF REPRESENTATIVES
New York 1 Timothy Bishop (D)
New York 13 Michael McMahon (D)
New York 19 John Hall (D)
New York 20 Scott Murphy (D)
GOVERNOR
Andrew Cuomo (D)
COMPTROLLER
Harry Wilson (R)
ATTORNEY GENERAL
Eric Schneiderman (D)
STATE SENATE DISTRICTS
11 Queens Tony Avella (D)
37 Westchester Bob Cohen (R)
40 Westchester, Putnam, Dutchess Michael Kaplowitz (D)
41 Dutchess, Columbia Didi Barrett (D)
NEW YORK CITY CHARTER
Question 1 No on reducing term limits from 3 to 2
Question 2 Yes to government reforms
CONNECTICUT
Poll hours: 6 a.m. to 8 p.m.
UNITED STATES SENATE
Richard Blumenthal (D)
HOUSE OF REPRESENTATIVES
Connecticut 4 Jim Himes (D)
GOVERNOR
Dan Malloy (D)
AROUND THE NATION
UNITED STATES SENATE
California, Barbara Boxer (D)
Colorado, Michael Bennet (D)
Florida, Charlie Crist (I)
Kentucky, Jack Conway (D)
Nevada, Harry Reid (D)
Pennsylvania, Joe Sestak (D)
Washington, Patty Murray (D)
Wisconsin, Russ Feingold (D)
HOUSE OF REPRESENTATIVES
Arizona 7, Raúl Grijalva (D)
Florida 24, Suzanne Kosmas (D)
Indiana 9, Baron Hill (D)
Iowa 1, Bruce Braley (D)
New Hampshire 1, Carol Shea-Porter (D)
Ohio 15, Mary Jo Kilroy (D)
Pennsylvania 8, Patrick Murphy (D)
South Carolina 5, John Spratt (D)
Virginia 5, Tom Perriello (D)
GOVERNORS
California, Jerry Brown (D)
Colorado, John Hickenlooper (D)
Florida, Alex Sink (D)
Georgia, Roy Barnes (D)
Ohio, Ted Strickland (D)
Massachusetts, Deval Patrick (D)
Texas, Bill White (D)
Rhode Island, Lincoln Chafee (I)

Records do speak for themselves.

Rovin on February 10, 2011 at 10:49 AM

This is not a big surprise at all. In the mainstream press, we often hear more about Republican mistakes than Democrats. For example, Charlie Rangel…kind of disappeared now, didn’t it? Selection bias is a big problem.

neoavatara on February 10, 2011 at 10:52 AM

Rovin on February 10, 2011 at 10:49 AM

correction: Frank Rich

Rovin on February 10, 2011 at 10:52 AM

St. Petersburg Times

A very liberal paper.

toliver on February 10, 2011 at 10:53 AM

I think Politifact staffers make an honest effort to fairly judge the accuracy of statements. I also think that they see the world through the lens of their personal belief, just as we all do. And since most reporters are liberal, they’ll tend to assume truths as given that a conservative might not.

hawksruleva on February 10, 2011 at 10:56 AM

Busted again. Great job.

paul1149 on February 10, 2011 at 10:56 AM

If Politifact was honest, fair and unbiased, it wouldn’t have won a Pulitzer.

cartooner on February 10, 2011 at 9:38 AM

Bingo. The progs long march through the awards committees across the globe has made all awards from Nobel to the Academy meaningless.

slickwillie2001 on February 10, 2011 at 11:22 AM

Did they ever fact check “You can keep your insurance”

antisocial on February 10, 2011 at 11:25 AM

I think Politifact staffers make an honest effort to fairly judge the accuracy of statements. I also think that they see the world through the lens of their personal belief, just as we all do. And since most reporters are liberal, they’ll tend to assume truths as given that a conservative might not.

hawksruleva on February 10, 2011 at 10:56 AM

So they are either stupid or corrupt. Not exactly respectable career choices.

slickwillie2001 on February 10, 2011 at 11:25 AM

Perhaps PolitiFact should be assigned a “Pants on Fire” rating.

GarandFan on February 10, 2011 at 10:20 AM

second that

tinkerthinker on February 10, 2011 at 11:30 AM

Can we please stop using the word “bias”? Bias assumes you didn’t really mean to do it.

“Corrupt” or “propaganda” is more accurate.

faraway on February 10, 2011 at 11:53 AM

Perhaps PolitiFact should be assigned a “Pants on Fire” rating.

GarandFan on February 10, 2011 at 10:20 AM

Winner

faraway on February 10, 2011 at 11:53 AM

As several have stated, the St. Petersburg Times is a far-left paper so it’s no wonder that PoliFact trends far left.

In 2009 it called “Death Panels” the “lie of the year” which is a matter of opinion. Yet, it overlooked proven lies by Obama such as “we’re going to have health care hearings on CSPAN,” “there will be no lobbyists in my administration” and “all bills will be posted on the Internet for five days before I sign them.”

bw222 on February 10, 2011 at 11:58 AM

I don’t know what the hubbub is about Politifact. It’s just another liberal blog that tries to hew to the center, like Politico or HuffPo (in the broad view), the way Fox or RealClearPolitics does on the right.

It gets cachet because conservative bloggers link it when they occasionally rebuke Obama. But even those scoldings are done gently and with reservation; witness yesterday when they analyzed his lie about not raising taxes, but dredged up the “Cadillac” slur on people who buy their own health insurance, identified the good causes that another tax was supposedly funding (which anyone who knows the first thing about the budget knows is false; governments cannot earmark money toward a cause, see: Social Security), apologized and spun for him, etc.

HitNRun on February 10, 2011 at 12:20 PM

Given politicians’ near universal proclivity for mendacity, D and R alike, a 39% to 12% ratio is fair evidence of selection bias in the choosing of statements to review.

I’m sure some members of the Society for Personality and Social Psychology can explain how this number doesn’t reflect bias, but only that Republican politician are vastly more ignorant and dishonest than their lefty comrade Democrats.

novaculus on February 10, 2011 at 12:38 PM

Nice to see Politifact getting fact-checked. A little taste of their own medicine.

Ever since they declared Palin’s notable quotable “death panels” the Lie of the Year after misrepresenting it, they’ve been dead to me.

/Oops! Is that uncivil rhetoric?

ornery_independent on February 10, 2011 at 12:41 PM

When Bush 41 said he wouldn’t raise taxes, and did, that was roundly considered the lie of the year at the time by everyone.

Well, Obama made the same claim running, and indeed raised taxes.

How on earth could this not merit lie of the year status?!?

18-1 on February 10, 2011 at 1:02 PM

“Smart Politics?” How smart do you have to be to know St. Pete Tunes is stark raving off the deep end progresso soup?

Now, we learn from careful analysis that a hard left biased evvaluation of political bias might be, uhhh, biased?

Stop the freakn presses…

swede7 on February 10, 2011 at 1:34 PM

Selection bias at Politifact?

The answer is obvious:

Politifact selects items to “fact check” based on input from its subscribers in the 57th state!

Since Conservatives, including many Republicans, cannot even find the 57th state on the map, it’s only natural that the selections are biased toward criticizing Republicans and Conservatives.

landlines on February 10, 2011 at 1:53 PM

PolitiFact is the offspring of the St Pete Times, so of course it’s biased.

Each time they call asking me to ssubscribe I answer in the negative. When asked why, I tell them they’re a liberal rag not fit to line the bottom of a bird cage. A few times the sales rep has repsonded that they hear that a lot. I’m surprised they’re still in business.

raz0r on February 10, 2011 at 2:59 PM

Any organization that can’t tell that Obama Care is a take over of the health care industry is not being honest with themselves probably is not qualified to be an arbiter of truth.

The Rock on February 10, 2011 at 4:40 PM

Here’s yet another Floridian who lives in the Tampa Bay area who can confirm that PolitiFact is nothing more than a propaganda instrument run by the state’s most left-leaning “news”paper, the St. Pete Times. Using the Times for wrapping fish would be an insult to the fish.

Kirin on February 10, 2011 at 5:37 PM

I live in the St Pete Times area, and believe me, Im glad someone is examining this totally partisan hack job Politifact. Its exactly on the mark that they are careful to choose certain questions that they think they can even remotely discredit. The Times subscriptions have fallen thru the floor and it amazes me that no starched suit over there is saying..’hey..we need to be more honest here’

malkinmania on February 10, 2011 at 5:58 PM