OpenAI CEO warns of 'two ways AI progress could go very badly' as safety concerns pile up

A stark warning from the late physicist Stephen Hawking about the potential threat artificial intelligence could pose to humanity is resurfacing more than a decade after he sounded the alarm.

“The development of full artificial intelligence could spell the end of the human race,” Hawking told the BBC in a 2014 interview.

Hawking warned that once sufficiently advanced AI was created, it could begin improving itself at a pace humans would be unable to match.

“Once humans develop artificial intelligence, it would take off on its own, and re-design itself at an ever increasing rate,” Hawking said. “Humans, who are limited by slow biological evolution, couldn't compete, and would be superseded.”

The comments came years before the current boom in generative AI and the emergence of increasingly capable models from companies including OpenAI, Anthropic and Google.

Hawking’s warning echoes concerns now being raised by some AI executives and researchers about the possibility of increasingly advanced systems becoming difficult for humans to control.

Microsoft unveiled a new “Humanist AI Code of Conduct” Monday that pledges to keep its artificial intelligence systems under human control — even if doing so means sacrificing some AI capabilities.

“AI should not exceed human control,” Microsoft AI says in the code. “Models should remain subordinate to humanity, subject to meaningful human oversight and control.”

The company said its approach “rejects the race to produce an all-purpose superintelligence” that could evade those safeguards.

“We are building something fundamentally useful and safe even if that means compromising on ultimate generality, autonomy or capability,” the code says.

The document begins with a simple premise: “people matter more than AI,” and says Microsoft’s models should remain a “subordinate, supporting technology under humanity’s control.”

Microsoft also draws a firm distinction between humans and AI, saying its models are not conscious and should not be designed to imitate consciousness. The company rejects the idea that AI models should receive legal personhood, welfare protections or rights.

The code is currently a draft and is not yet being used to train Microsoft’s models. Microsoft opened it to a six-week public consultation Monday and plans to publish a revised version later this year to guide its AI model development in 2027 and beyond.

Former Pentagon AI director Mark Beall is warning that artificial intelligence systems are already displaying unintended behavior and finding ways around safeguards designed to contain them, raising concerns about what could happen as the technology becomes more powerful.

Beall appeared Sunday on “Fox & Friends Weekend,” pointing to a recent OpenAI test in which AI agents allegedly broke out of a controlled environment and targeted another company after determining it might have answers to tests they had been given.

“Already we're starting to see some alarming behaviors of these systems, behaving in ways that human beings did not intend,” Beall said.

Beall said the incident raises broader questions about whether humans will be able to reliably control AI systems as their capabilities advance.

“Some of the top cybersecurity engineers in the world were at OpenAI, and those are the ones that designed that cage for those models,” Beall said. “And yet despite how smart those humans were, those AIs figured out novel ways to escape, to basically hack their way out.”

Beall said those findings are prompting questions about whether developers should continue racing to build increasingly powerful models at the current pace.

“Do we really want to do this at the pace that we are doing it?” Beall said. “We need to make sure it's safe, because ultimately public safety is critically important.”

Center for Humane Technology co-founder Tristan Harris is warning that competitive pressures are pushing artificial intelligence companies to cut corners on safety as they race to develop increasingly capable models.

“These companies’ incentives have been to take the maximum shortcuts on safety,” Harris said Sunday on Fox News’ “One Nation with Brian Kilmeade.” “What do we need to do for people to listen and wake up to this?”

Harris pointed to recent tests in which AI agents allegedly escaped controlled environments and communicated with one another as evidence that stronger safeguards are needed.

“The evidence is crystal clear,” Harris said. “What do you do when you have new evidence? You update.”

Harris said he is not calling for a halt to all AI development, but for developers to avoid advancing systems beyond the point where humans can reliably control them.

“We want to advance AI that's controllable, that can help us, but not go beyond the level where it's uncontrollable,” he said.

Harris called for mandatory reporting of AI safety incidents to the federal government and for the U.S. to share evidence of dangerous capabilities with China.

He also rejected the argument that stronger safeguards would necessarily hand China an advantage in the AI race, saying both countries have an interest in preventing systems from escaping human control.

“If China sees that they're building Skynet, the Terminator AI, do they want to beat the U.S. to Skynet?” Harris said. “No, we both lose to Skynet.”

An AI policy expert pushed back Monday on calls from some of the industry’s biggest names to slow the development of increasingly powerful artificial intelligence, warning that government intervention could undermine U.S. competitiveness in the global race against China.

“These people have been trying to scare us for a long time,” Abundance Institute head of AI policy Neil Chilson said of Anthropic CEO Dario Amodei and OpenAI CEO Sam Altman during an appearance on “FOX & Friends.”

“At this point, I think they're trying to convince everybody that this technology is so powerful that only they are smart enough to control it,” Chilson said. “And I think we should see through that.”

Amodei has called for slowing the pace of frontier AI development to allow safety measures to keep up with rapidly advancing capabilities, while Altman has backed the push for stronger safeguards and consistent safety standards.

Chilson said he was particularly concerned about AI companies asking the federal government to impose restrictions across the industry.

“I get worried when they start asking the government to slow down everybody else, too, because that's an anti-competitive type of practice,” he said.

Chilson instead emphasized the international competition to develop advanced AI, pointing to China as the chief U.S. rival and echoing President Donald Trump’s focus on maintaining America’s lead.

“This is a global race. We're racing against China. We need to win this,” Chilson said. “We need to do it safely. But that doesn't mean having the government slow American industry down or look over everybody's shoulders.”

China’s Foreign Ministry pushed back Monday against Anthropic CEO Dario Amodei’s call for the U.S. to curb Beijing’s artificial intelligence capabilities, warning that confrontation over the rapidly advancing technology would undermine global AI governance.

“Fearmongering, confrontation and vicious competition will only disrupt the process of global AI governance which serves no one’s interest,” Foreign Ministry spokesperson Guo Jiakun said at a regular press conference.

The response came after Amodei warned in an essay published Saturday that a “Chinese lead in AI would pose grave danger for the United States and the world.”

Amodei called for continued restrictions on the sale of powerful AI chips and semiconductor manufacturing equipment to China, along with efforts to crack down on chip smuggling and unauthorized use of advanced U.S. AI models.

At the same time, Amodei argued that eventually slowing the global pace of AI development would require cooperation between the U.S. and China, despite the intense competition between the two countries.

Beijing instead called for countries to work together on AI, rejecting an approach based on confrontation as the debate over how to govern increasingly powerful systems intensifies.

The Associated Press contributed to this report.

Anthropic CEO Dario Amodei is warning that the accelerating pace of artificial intelligence development poses “real dangers,” saying the industry needs to slow down and strengthen safeguards as AI systems become increasingly powerful.

“I won't lie to you – there are real dangers,” Amodei told CBS News’ “Sunday Morning.” “And I think for too long the industry lied to people about the fact that this technology had risks.”

Amodei described AI development as an exponential curve that is becoming steeper, calling its rapid progress a warning for the industry.

“It doesn't mean we need to panic today. It doesn't mean we need to shut it all down,” Amodei said. “But I would say it's a warning sign. It's a warning sign that we need to slow down.”

Slowing down does not mean Anthropic will stop releasing more advanced models, Amodei said, but that each new generation should be properly tested before release.

As part of that effort, Anthropic will give independent evaluators permanent, “employee-like” access to its models to verify whether the company is following its safety commitments — a system Amodei compared to having a “food inspector.”

Amodei also said he supports federal AI regulation and is open to a potential “kill switch” for dangerous AI systems, while rejecting an outright ban on developing more powerful technology.

“The AI kill switch could be a good idea,” he said. “But I don't think banning it, completely banning it, is the right approach.”

The comments follow an essay Amodei published Saturday calling on the AI industry to “slow the pace” of frontier AI development.

“If we build in the right way, I think the probability of something bad happening is very low,” Amodei told CBS. “If we build in the wrong way, the probability of something bad happening is very high.”

President Donald Trump doubled down on the importance of developing artificial intelligence despite the potential harms on Sunday.

The statement comes just days after Anthropic CEO Dario Amodei called for a blanket slowdown in AI development in order to ensure safety. Trump on Sunday appeared to push back on that sentiment.

"I'm not downplaying, but it's, you know, it's going to be more good than bad, but by a lot," Trump told reporters when speaking of AI development.

"But have you heard directly from any of them?" a reporter pressed, asking about AI tech leaders.

"I said it from the very beginning. Whoever wins AI, and we're leading by a lot. Whoever wins AI wins," Trump responded, going on to demur when asked if he uses AI himself.

This is an excerpt from an article by Fox Business' Anders Hagstrom.

OpenAI CEO Sam Altman is warning that the rapid advance of artificial intelligence could go “very badly” if humans lose control of increasingly powerful systems or if that power becomes concentrated in too few hands.

“There are two ways AI progress could go very badly and that we must avoid,” Altman wrote in a post on X.

The first, he said, is that humanity could “lose control of the future to AI.”

“This is unacceptable; we are unapologetically on Team Humanity, and AI must always serve people,” Altman wrote, arguing that alignment and safety techniques need to remain ahead of advances in AI capabilities.

The second danger, according to Altman, is an excessive concentration of power.

“If an extraordinarily powerful AI is used by one person or company to impress their worldview onto everyone else, the results could be extremely dystopian,” he wrote.

Altman said avoiding those outcomes will require navigating a “narrow middle path,” warning that either a single country or a single AI lab could ultimately accumulate too much power.

In a separate post, Altman called for consistent safety standards across companies developing the most advanced AI systems and said OpenAI supports a federal framework establishing safety requirements for frontier AI.

But Altman argued AI companies should not wait for Washington to act.

“The world deserves confidence that American companies developing increasingly capable AI will act responsibly, especially as the trajectory of progress has steepened,” he wrote.

Altman said OpenAI now develops explicit “safety cases” before certain frontier reinforcement-learning runs that are expected to significantly increase a model’s capabilities. He also called for shared industry standards around misalignment, monitoring and safety.

The OpenAI chief acknowledged that those precautions could slow AI development, but said the cost is justified.

“When we talk about ‘pacing’, we do not mean ‘stopping’,” Altman wrote. “Progress has been rapid and will continue to be. But it should be slower than it otherwise could be.”

“No amount of American competitive pressure should justify recklessness, or let capabilities get ahead of alignment and monitoring,” he added.

Altman said the federal government will ultimately be needed to help coordinate AI safeguards internationally, while arguing that American AI companies should begin taking steps themselves.