While it’s true that the number of nonbelievers is the United States is growing, it’s still small—roughly 3 percent of U.S. adults self-identify as atheists. And while more and more Americans say they’re not part of any particular religion, they’ve historically been in good company: At the end of the 19th century, Schmidt estimated, around a tenth of Americans may have been unaffiliated from any church or religious institution.

As the visibility and number of American atheists has changed over time, the group has gone through its own struggles over identity. Even today, atheists are significantly more likely to be white, male, and highly educated than the rest of the population, a demographic fact perhaps tied to the long legacy of misogyny and marginalization of women within the movement. At times, nonbelievers have advocated on behalf of minority religious rights and defended immigrants. But they’ve also been among the most vocal American nativists, rallying against Mormons, Catholics, and evangelical Protestants alike.

Schmidt and I discussed the history of atheists in the United States, from the suspicion directed toward them to the suspicions they have cast on others. Our conversation has been edited and condensed for clarity.