Massachusetts Gov. Deval Patrick (D), Sen. Sheldon Whitehouse (D-R.I.) and California Attorney Gen. Kamala Harris each bowed out of the running within hours of Holder’s announced resignation Thursday – all saying they were committed to their current jobs.

Several other names are swirling around Washington’s rumor mill, but experts say any would-be successor – especially those with higher political aspirations – may have serious reservations about taking the Justice Department’s helm at the tail end of Obama’s presidency.

“The attorney general tends to get, by virtue of the office, caught up in very controversial decisions, actions, and activities,” said Brookings Institution fellow John Hudak. “Somebody who has real ambitions down the road would like to avoid having to make decisions that are unpopular with a part of the population down the road, or carrying water for the president for a choice that isn’t to their advantage.”