Omohundro explained, “Okay, let’s say it just played its best possible game of chess. The game is over. Now comes the moment when it’s about to turn itself off. This is a very serious event from its perspective because it can’t turn itself back on. So it wants to be sure things are the way it thinks they are. In particular it will wonder, ‘Did I really play that game? What if somebody tricked me? What if I didn’t play the game? What if I am in a simulation?’”

What if I am in a simulation? That’s one far-out chess-playing robot. But with self-awareness comes self-protection and a little paranoia.

Omohundro went on, “Maybe it thinks it should devote some resources to figuring out these questions about the nature of reality before it takes this drastic step of shutting itself off. Barring some instruction that says don’t do this, it might decide it’s worth using a lot of resources to decide if this is the right moment.”

“How much is a lot of resources?” I asked.

Omohundro’s face clouded, but just for a second.

“It might decide it’s worth using all the resources of humanity.”