Polls don't measure what you think they measure

But problems start to multiply when you get far beyond these sorts of queries. Most well-known is the phenomenon of social-desirability bias, where survey respondents will tend to answer questions in ways that would make themselves look better. One rather amusing example of this: Men between the ages of 25 and 44 report having had an average of 6.1 opposite-sex sexual partners; women in the same age group report an average of 4.2 partners. Supposing that there are roughly an equal number of sexually active men and women, this strongly suggests that someone is lying. Another example: The South is generally considered to be the American region with the greatest obesity problem, using self-reported data for height and weight. But a University of Alabama study used “direct measures” — weighing people — and found that the upper Midwest actually had higher obesity rates. The South may simply have less of a stigma against obesity, encouraging relatively honest self-reporting.

But there’s more going on than just social-desirability bias. Consider church attendance. Most polls suggest that about 40 percent of the American population attends church or synagogue more than once a week. That figure has been roughly stable since at least the late 1930s, reaching a high of 49 percent in the mid 1950s, and a low of 36 percent today (although it reached 37 percent in 1940, and 38 percent in the mid 1990s). Yet, over the last 50 years, a number of denominations have been losing members, and a 1993 study found that actual attendance rates were about half the self-reported numbers for both Protestants and Catholics. The study tried pinning this on social-desirability bias, but this doesn’t really make sense. Views on church attendance were far more conservative in the 1940s and the 1950s than in the 1990s, yet it seems that more people were misrepresenting their church habits in the 1990s, which runs exactly counter to the expected trend.