“There are so many communities where you still have persistent patterns of segregation,” said Tom Perez, the Labor secretary and the former head of the Justice Department’s Civil Rights division. “It leads to a lack of understanding and that is unfortunate, and that can have ill consequences.”

And African American and civil rights leaders say this tension has been heightened by conservatives’ opposition to Obama they believe has been waged on racial grounds, along with measures like voter ID laws they say are meant to keep minorities from the polls.

No one expected Obama to change 400 years of legacy in a term and a half as president. But Obama took a very cautious approach in his first term, largely avoiding the topic, leading black leaders to complain that they were watching their gains erode by a president who took their support for granted. The outrage that caught the White House by surprise when he said police “acted stupidly” in Henry Louis Gates’ 2009 arrest only helped solidify the West Wing’s hands-off approach.

“He wanted to be perceived in a color-blind way, but the way he went about that was giving short shrift to African-Americans so that he wouldn’t be perceived as hung up on race,” said Paul Butler, a former DOJ prosecutor who’s now a professor at Georgetown Law. “One of the problems for people who don’t want to be perceived as hung up on race is that they end up obsessed with it.”