“When roboticists are asked by nervous onlookers about safety, a common answer is ‘We can always unplug it!’ But imagine this outcome from the chess robot’s point of view,” writes Omohundro. “A future in which it is unplugged is a future in which it cannot play or win any games of chess. This has very low utility and so expected utility maximisation will cause the creation of the instrumental subgoal of preventing itself from being unplugged. If the system believes the roboticist will persist in trying to unplug it, it will be motivated to develop the subgoal of permanently stopping the roboticist,” he writes.

In other words, the more logical the robot, the more likely it is to fight you to the death.

The problem of an artificial intelligence relentlessly pursuing its own goals to the obvious exclusion of every human consideration is sometimes called runaway AI.

The best solution, he says, is to slow down in our building and designing of AI systems, take a layered approach, similar to the way that ancient builders used wood scaffolds to support arches under construction and only remove the scaffold when the arch is complete.