If you bracket the science fiction fear of an artificial intelligence that consciously tries to overthrow or replace its creators — and I’ll return to that scenario — the real fear shouldn’t be about the robots, but about humans.
Consider past human societies that have had ready access to subjugated labor — like the slaveocracy of the antebellum American south. Imagine that, instead of slaves, the planters had robots working in their homes and fields. Just as it was difficult for free labor to compete with slave labor, it would be difficult for free labor to compete with robots. If the imagination of the planter class was limited to living the good life as it could be supported by their robots, and if the poor humans who owned no robots could be prevented from getting them (say, by gun-toting police robots owned by the planters), then yes, one can imagine the free, non-robot-owning humans becoming progressively more improverished and debased until finally they were rendered figuratively and then literally extinct. We don’t need a robot apocalypse, though, to teach us that one group of humans can and sometimes will choose to wipe out another group that it finds inconvenient.