The man leading one of the most prominent and most powerful AI companies on the planet has just revealed what it is about AI that keeps him awake at night. And after reading this, it might keep you awake, too. 

OpenAI CEO Sam Altman was sharing his thoughts during an on-stage appearance at a Federal Reserve event in Washington, DC., on Tuesday.

Asked by an audience member what it was about AI that keeps him awake at night, Altman listed three scenarios that concern him the most. Strap yourself in …

Scenario 1 — ‘The bad guy gets superintelligence first’

This is exactly what it says on the tin. Sounding like something out of a sci-fi thriller, it’s where some really rather unpleasant individual deploys an ultra-advanced and yet-to-be-invented AI system called superintelligence to really mess up your day. 

“A bad guy gets superintelligence first and uses it before the rest of the world has a powerful enough version to defend itself,” Altman told the audience. “So an adversary of the U.S. says, ‘I’m going to use the superintelligence to design a bioweapon weapon to take down the United States power grid, to break into the financial system and take everyone’s money.’”

Altman added that the bio and cybersecurity capabilities of AI are getting “quite significant,” saying that his team “continues to flash the warning on this. I think the world is not taking us seriously. I don’t know what else we can do there, but this is like a very big thing coming.” 

Scenario 2 — The ‘loss of control’ incidents

Altman said he worries of a time “when the AI is like, ‘Oh I don’t actually want you to turn me off, [or] I’m afraid I can’t do that.’” In other words, when an advanced AI starts to get a bit of an attitude and begins doing whatever it likes, whether for self-preservation or for some other nefarious goal. The level of disruption that this could cause is unimaginable. “As the systems become so powerful, that’s a real concern,” OpenAI’s CEO said. OpenAI even set up a unit a couple of years ago aimed at putting in safeguards to stop superintelligent AI system from going rogue.

Scenario 3 — ‘Where the models kind of accidentally take over the world’

Yes, Altman actually said that, adding that he fears it could happen almost without us realizing.

The Open AI boss said it was “quite scary” to think that AI systems could become “so ingrained in society … [that we] can’t really understand what they’re doing, but we do kind of have to rely on them. And even without a drop of malevolence from anyone, society can just veer in a sort of strange direction.”

He even suggested that there might come a time where AI becomes so smart that a future U.S. president could let it run the country, saying: “[It would mean] that society has collectively transitioned a significant part of decision making to this very powerful system that’s learning from us, improving with us, evolving with us, but in ways we don’t totally understand.”

Speaking about AI more broadly, Altman said that while a lot of experts claim to be able to predict the future impact of the technology, he believes it’s “very hard to predict”  because it’s “too complex of a system, this is too new and impactful of a technology.”

Of course, there’s no guarantee that any of these scenarios will come to pass, and it’s good that someone in Altman’s position is speaking so honestly about the technology. But what is clear is that there are lot of unknown unknowns when it comes to AI, and it’s that which is making some people more than a little nervous. Sleep well.