The first problem with its elimination is purely pragmatic. Electing presidents via popular vote would be a logistical disaster, rendering every recount a national recount. Moreover, eliminating the Electoral College would require a wholesale revision of the Constitution. That process would almost surely fail, and it would certainly be ugly.
Reasonable reformers understand this, which is why they propose a national compact by which states agree to direct their electors to vote in accordance with the national popular vote.
That would be better than outright repeal, partly because if the compact backfired, it could be easily reversed. But popular elections would still raise problems. Candidates would be incentivized to rack up huge majorities among their bases. An outright majority of votes could be gotten simply with populist appeals to a handful of large, highly urbanized states. If you’re a pure-democracy fetishist, that may sound fine. But how would it lessen polarization?
Indeed, I think polarization, not love of democracy, is what’s driving antipathy for the Electoral College.