It’s not clear how this will play out. Vin Weber, a top Republican strategist and former congressman, said, “The big question confronting the American system after this election is: Will there be an open, conservative Republican Party and a left-of-center, collectivist Democratic Party, or also a right-wing authoritarian party?” (The hope that Mr. Trump’s selection of Gov. Mike Pence of Indiana as his running mate signaled a new turn and tone was undercut by the way the nominee introduced his vice-presidential candidate.)

Some argue that the problem could become even more acute if Mr. Trump wins the presidency and then tries to reshape the party in his mold: protectionist, anti-immigration, isolationist, more populist on economics and only selectively supportive of Wall Street and financial markets. If so, how much pushback would there be from Republican traditionalists and movement conservatives? “It would be four years of upsetting the apple cart most every day,” said Fred Davis, a Republican strategist.

If Mr. Trump loses — which many Republicans expect, despite narrowing poll numbers — the dynamics change but may be equally complicated.