With Attorney General Eric Holder planning his exit from the national stage, Americans are starting to look back on his tenure, assess what his legacy will be, and determine his place in history. The longer White House officials (and presidents, for that matter) are out of the limelight, the more the public’s estimation of them improves. When they leave office, White House officials are often at the nadir of their popularity. That is good news for Eric Holder. As of today, he is only the third most reviled former or current Obama administration official.

According to a recent YouGov poll, only 26 percent of Americans have a favorable view of the attorney general. 37 percent have an unfavorable opinion of Holder after his six years at the head of the Justice Department.

But that’s a pessimist’s view of Holder’s dismal popularity rating. One could just as easily look at those figures and note that Holder is not as unpopular as two other former Cabinet officials. As of April, only 19 percent had a favorable opinion of former Health and Human Services Sec. Kathleen Sibelius while 41 percent had a negative opinion of her. When Secretary of Veterans Affairs Eric Shinseki stepped down, only 18 percent of the public viewed him favorably while 40 percent had the opposite opinion.

That might be cold comfort for Holder. Both Shinseki and Sebelius stepped down amid or following controversy. Holder has presided over a controversial Department of Justice; it has failed to thoroughly investigate the Fast and Furious gun-walking scandal, the IRS’s targeting of conservative groups, or his investigators’ decision to monitor the activities of journalists. Each of these scandals has taken a toll on the public’s belief that Holder served as an impartial administrator of the law.

It is not merely conservatives who are down on Holder. Liberals, too, are disappointed with the attorney general’s handling of the nation’s legal affairs. “Liberals have expressed disappointment with Holder’s handling of some issues, particularly the failure to prosecute bankers involved in the 2007-08 financial crisis,” YouGov noted. But most Democrats are not moved to despise Holder merely because he did not force the nation’s financial professionals into the stockades in 2009. 40 percent of Democrats surveyed had no opinion on the attorney general at all.

The attorney general has also been famously outspoken on issues relating to race. He has on multiple occasions insinuated that the opposition he and the president experienced from Republicans was due, in part, to racial antipathy. Holder has also asserted that America is a “nation of cowards” on the issue relating to race. This has taken a toll on Holder’s appeal among white survey respondents (42 to 21 percent view him unfavorably), but it has endeared him to African-Americans (57 to 6 percent view him favorably).

It is probably too early to be sure what Holder’s legacy will be. It seems a certainty, however, that posterity will see him as a divisive figure that created lasting partisan divisions over the course of his tenure. It will be a legacy with which many members of this administration will have to contend.