Sam Altman calls an AI warning at a moment when the people who build the technology, rather than those who regulate it, are setting the terms of the debate. In an overnight post on X, the OpenAI chief executive described two ways AI progress could go badly wrong. The first is humanity losing control of the future to AI, which he called unacceptable. Next comes power gathering in too few hands, whether one person, one company, or one country.
Altman wrote that OpenAI is “unapologetically on Team Humanity” and AI must serve people. Keeping that promise, he argued, requires alignment and safety techniques to stay ahead of gains in model capability. On the second risk, he described a dystopian outcome if one person or company used an extraordinarily powerful system to press its worldview on everyone else.
A warning shaped by a rival’s essay
The timing was not accidental. Dario Amodei, chief executive of Anthropic, published an essay on Saturday titled “We Must Pace the Frontier.” The Dario Amodei essay argues the industry should slow the rate at which it improves model capabilities so risk prevention has time to keep up. He was careful to say pacing does not mean halting training or technical progress.
The essay sets out three steps. Anthropic has committed to the first on its own: permanent, employee-level access for third-party evaluators who can verify safety measures, report incidents, and assess model alignment during training. Step two asks frontier companies in democracies to agree on common safety standards and limits on development speed, ideally through legislation. The third requires agreements with non-democratic governments.
Altman answered within hours. He agreed on the need to pace the frontier and committed the company to the same evaluator access. Elon Musk replied with “Dario is right.” Google DeepMind’s Demis Hassabis said the direction was on the right track.
After Sam Altman calls an AI warning, OpenAI rewrites its safety process
The post in which Sam Altman calls for an AI warning carried operational detail alongside principle. Altman said earlier safety tools, such as Preparedness Frameworks and Responsible Scaling Policies, focused on the deployment of finished models. The focus has to shift to development itself, he argued. OpenAI now prepares explicit safety cases before large reinforcement learning runs expected to raise a model’s capabilities substantially, on top of the review it already conducts before release.
He also backed a federal framework setting consistent safety rules for frontier AI, and said companies need not wait for legislation. This is the governance tension at the centre of the story. The labs are writing their own rules and inviting outsiders to check the work. Whether anyone can compel them to keep those rules is a separate matter, and Stability AI founder Emad Mostaque has publicly questioned whether evaluators without enforcement powers can hold a lab to its commitments.
Sam Altman calls an AI warning days after a researcher walked out of Anthropic saying much the same in blunter terms. Jacob Coxon, who spent three years on pretraining research at OpenAI and then Anthropic, resigned on September 8. He wrote on X that neither company is acting responsibly, and accused both of racing toward self-improving superintelligence while “gambling with our lives.”
Evan Hubinger, an alignment science lead at Anthropic, replied that Coxon was correct. He put the chance of AI killing all humans within the next decade above 10 per cent. The risk from current models, he stressed, is low. Anthropic, he added, does not yet have a plan to solve alignment for superintelligence.
Where governments stand as Sam Altman calls an AI warning
President Trump rejected the case for slowing down. Speaking to reporters at his golf resort in Doonbeg, Ireland, over the weekend, he said the United States leads China in AI. He wants to keep it that way, he added, because “whoever wins AI wins.” The federal government could set guardrails, he allowed, though he described the warnings as coming from negative forces raising things that will not happen.
Amodei’s essay shares the concern about Beijing. It argues a Chinese lead in AI would pose grave danger and urges tighter limits on which chips reach China. On Sunday, Chinese leader Xi Jinping made his own pitch to Global South leaders in India. He called for openness and collaboration and revived a proposal for a global AI governance pact, without giving details.
AI regulation is now contested on three fronts. The labs favour rules they help write. Washington favours speed. Beijing favours openness on its own terms. Xi is expected to visit the United States later this month, with AI safety on the agenda. Sam Altman calls for an AI warning that Washington, for now, does not echo. Whether the question of who could lose control of AI, and who answers for it, reaches that table is the next test.
