Five Republican senators raised the stakes around Justice Stephen Breyer’s retirement, telling CNN they’d oppose any likely nominee out of this White House.

But Democrats in the Senate and the White House, though they’ve come off the November elections more worried in recent weeks about a potential wipeout in the midterms, are still avoiding calling directly for Breyer to quit, fearing that it would backfire and encourage the 83-year-old to stay on the bench.

Breyer has told several people who’ve made unofficial efforts to push him to retire that he thinks the confirmation process shouldn’t be political, according to people told of those discussions, and Democrats worry he’d remain as an act of resistance to show he’s not bowing to politics.

Still, top Democrats across Washington would like Breyer to announce he’s going even before the end of the court term in June, so they can get moving on confirmation hearings well before the midterms.