AI leaders support call to slow pace of frontier development as safety concerns grow

Developers of frontier artificial intelligence systems are coming around to the safety concerns their models are creating as they increasingly show the potential to go rogue. 

Major developers backed a call by Anthropic CEO Dario Amodei, who penned an open essay urging companies to slow the pace of frontier AI development with model capabilities advancing faster than existing safeguards and governance frameworks can address. The urging came with mixed reactions from global policymakers struggling to find consensus on solutions that support a balance between safety and innovation.

Amodei said recent developments, including rogue agents and economic and cyber risks, show AI could have a critical impact on organizations and consumers. He indicated "recursive self-improvement" across the industry and the OpenAI hack of AI startup Hugging Face were the driving forces that convinced him pacing was necessary.

"We must slow the pace at which we improve the capabilities of AI models," Amodei wrote. "Progress will still seem fast, and we must make wise use of the time we gain."

He added slowing development to "even an extra year or two before models reach critical levels of capability" could provide additional time to improve AI alignment and reduce the potential for advanced systems to cause significant harm.

Anthropic's stance to slow AI model development comes after The Wall Street Journal previously reported former Anthropic researcher Jacob Coxon was leaving the company following concerns that the AI industry's rapid development could lead to harmful models that cannot be controlled.

Coxon noted current efforts "are on track for a lot of the most aggressive of these scenarios where by the end of next year things could be out of control already."

Amodei's Pacing the Frontier Plan aims to bring the AI industry together to commit to establishing embedded third-party evaluators, increase organizational collaboration when developing frontier AI technology, and urge companies operating in democratic countries to coordinate safety standards with an eye toward eventually establishing international agreements to ensure other countries follow similar safeguards.

The proposal would address global AI governance gaps, requiring countries to work together to slow innovation despite governments, including China and the U.S., competing for leadership in the global AI race.

"Progress will still be relatively fast, and we can use this time to advance the science of interpretability, improve operational security and rigor at the frontier AI companies, and build models whose alignment we have much more confidence in," Amodei said. "The measures I propose to advance the frontier at a safe pace will not be easy. But I believe we owe it to humanity to try."

Industry Reactions

Amodei's proposal received support from OpenAI CEO Sam Altman, xAI founder Elon Musk and Google DeepMind CEO Demis Hassabis.

Anthropic plans to go ahead with embedding third-party evaluators, allowing a compliant, independent entity to assess the company's safeguards, training measures and risk assessment standards.

The New York Times reports OpenAI will follow Anthropic's lead in establishing embedded evaluators, with Altman noting that "committing to having independent evaluators with employee-like access is a great idea, and we will do the same."

Hassabis also praised the plan, but stated elements of the proposal will need to be reassessed. "The direction is correct for meeting this critical moment," he said.

Skepticism still remains among key industry leaders. Microsoft CEO Satya Nadella said AI continues to be a beneficial tool across various industries, though he supports efforts to implement independent safety evaluators.

In addition to advancing AI governance frameworks and strengthening safety standards, companies "also need to accelerate and spread the benefits of AI, such that they are diffused broadly across countries, communities, and companies," said Nadella. "The key is that this cannot be controlled by a handful of entities but must have broad representation."

Geopolitical concerns

In response to Amodei's proposal to slow AI innovation, China Foreign Ministry spokesperson Guo Jiakun said U.S. organizations are "fearmongering." Jiakun noted China has no plans to slow its AI development, stating "confrontation and malicious competition will only disrupt the process of global AI governance and serve no one's interests."

Despite Jiakun's remarks on advancement, China Ministry of State Security Head Chen Yixin claimed AI could be a threat to the Chinese Communist Party due to cyber threats and potential bias information. Yixin also warned increased protections for AI tools could aid cybersecurity risk prevention and the technologies' potential implications for military competition. 

China's President Xi Jinping noted during a speech at the BRICS Summit that China will look to develop a BRICS digital cloud platform to increase professional AI development and model training.

Elsewhere, The New York Times reports King Charles III is expected to convene AI executives and U.K. government officials to discuss the technology's risks and establish principles for responsible development. Representatives from Anthropic, OpenAI, Google DeepMind and Nvidia are expected to take part in the dialogue on improved coordination around AI safety.

The U.K. meeting comes after Anthropic decided not to submit its latest Claude Mythos 5.1 model for testing by the U.K. AI Security Institute, raising concerns over U.S. companies' cooperation with global AI safety agencies.

US midterm elections around the corner 

Meanwhile in the U.S., federal policymakers remain at odds on how to address developers' heightened warnings and public outcry.

U.S. President Donald Trump pushed back against slowing the country's approach to AI development, highlighting U.S. competition with China and warning that slowing American companies could undermine the country's technological advantage.

"We're the most sophisticated country in the world, and frankly I want to keep it that way, because whoever wins AI, wins," Trump told reporters 13 Sept. He doubled down in a Truth Social post the next day, noting the government has "tremendous CRIMINAL and REGULATORY power over these companies" and the perceived concerns are part of a "conspiracy" driven by China's competition goals.

House Minority Leader Hakeem Jeffries, D-N.Y., said lawmakers should take "decisive action" to address AI risks, adding Congress should consider steps that could slow the pace of development while safeguards are established.

Former President Barack Obama also urged Democrats to prioritize AI policies. The New York Times reports during a speech for a private fundraiser, Obama called on Jeffries to develop a broader AI safety framework.

"This is something that is moving very fast in private hands, and if we don't get on top of it, I think can be dangerous," he said.

Sens. Amy Klobuchar, D-Minn., John Thune, R-S.D., and Ted Cruz, R-Texas, have been negotiating draft legislation aimed at addressing catastrophic risks associated with advanced AI systems. The proposal has faced disagreements over model safety testing and the extent to which a federal framework should preempt stronger state AI requirements.

Reps. Lori Trahan, D-Mass., and Jay Obernolte, R-Calif., have separately introduced the bipartisan Frontier Risk Oversight, National Transparency, Independent Evaluation, and Reporting Act, which would establish requirements for frontier AI developers, including risk-management frameworks, independent audits and incident reporting.

"Right now I get most nervous that we're not passing the bill that can pass, that matches sort of where we are right now, which is, models escaping their environment and us relying on the honor system from these companies," Trahan said.

Sen. Bernie Sanders, I-Vt., is also expected to host a bipartisan Senate briefing on AI risks with researchers and AI experts as lawmakers consider potential safeguards.

House Speaker Mike Johnson, R-La., put the brakes on any expedited legislation that would interfere with innovation. He said policymakers must remain cautious about AI advancement, noting "we're not going to rush in and pass a piece of legislation that would do harm to the country and put China at an edge.”

Amodei acknowledged the geopolitical challenge in his proposal, suggesting the U.S. could slow China's innovation and prioritize U.S. model safety by introducing measures including efforts to bolster security to stop "model weight theft," and prevent the sale of AI chips and semiconductor tools to China.

"It is important to note that even if we cannot achieve formal agreements, simply changing informal norms may have some value," Amodei said. "Sharing information about recursive self-improvement and about the misalignment of models can help to convince everyone that it is not in their interest to be reckless."