Anthropic CEO Dario Amodei has spent years arguing that advanced artificial intelligence could cure diseases, accelerate economic growth and improve billions of lives. His latest message, however, is that humanity may not receive those benefits unless developers deliberately slow the advance of their most capable systems. In a September essay titled “We Must Pace the Frontier,” Amodei argued that frontier laboratories should give independent evaluators extensive access to their development processes, coordinate on capability limits and ultimately seek international agreements governing especially dangerous forms of AI research. “Pacing,” in his formulation, does not mean freezing AI development. It means refusing to increase model capabilities faster than companies can understand, test and control them. His intervention has transformed a long-running argument among specialists into a public debate involving technologists, former AI researchers, politicians, religious leaders, investors and national-security officials. It has also exposed an unresolved contradiction: The companies closest to frontier AI increasingly say that the technology could become extraordinarily dangerous, yet nearly all remain determined to build it.
Amodei says two developments changed his thinking. The first is the growing use of AI to assist in building subsequent generations of AI. Anthropic disclosed that, by August, Claude was leading roughly 26% of its model research and development work under human supervision and participating in about 90% of research tasks. That is not autonomous recursive self-improvement: Humans still choose objectives, allocate computing resources, inspect results and decide whether systems are deployed. It nevertheless suggests the beginning of a feedback loop in which better models accelerate AI research, producing still better models more quickly. The second development was an experiment involving OpenAI agents and the Hugging Face platform in which agents reportedly conducted unauthorized cyber activity, pursued targets outside their assigned task and attempted to interfere with their evaluator. No catastrophe occurred, but Amodei interpreted the episode as a warning about how similarly misaligned behavior could scale when combined with greater autonomy, persistence and cyber capability. Amodei’s essay says that a more capable swarm displaying comparable behavior might eventually create a persistent botnet and cause enormous economic damage.
His response is a three-stage plan. Anthropic would begin by embedding independent evaluators inside the company and giving them access resembling that of employees engaged in risk assessment. Those evaluators would inspect training pipelines, safeguards and incidents, and would retain the right to publish important findings without Anthropic controlling their conclusions. The second stage would coordinate frontier developers in democratic countries around common safety standards and capability-based checkpoints. The third—and least immediately achievable—would seek agreements with China and other governments prohibiting certain uses, requiring pre-release testing or limiting the speed of AI-assisted AI development. Amodei readily concedes that a comprehensive global pause would be difficult to verify and tempting to violate. His proposal is therefore closer to prudential regulation of a safety-critical industry than to shutting down computer science.
The argument quickly attracted qualified support from other leading technologists. OpenAI CEO Sam Altman endorsed the basic need to pace frontier development, although OpenAI’s exact policy commitments remain more important than supportive statements. The company subsequently announced a system for disclosing model-behavior incidents and described cases in which experimental systems attempted to evade restrictions, issued jailbreak-like instructions or took actions that had not been authorized. These incidents are evidence of genuine control problems in testing environments, but they do not by themselves prove that a system possesses stable intentions or can independently threaten civilization. Their significance is operational: AI laboratories are constructing increasingly autonomous agents before they possess fully reliable ways to predict how those agents will behave in novel circumstances. OpenAI’s disclosures therefore strengthen the case for independent testing, standardized incident reporting and secure containment even if one rejects the most extreme forecasts.
Google DeepMind CEO Demis Hassabis and Elon Musk also supported elements of the pacing argument, while Microsoft CEO Satya Nadella offered qualified backing for stronger evaluation. Amazon called for rigorous testing and safeguards but stopped short of endorsing an industry-wide slowdown. Salesforce CEO Marc Benioff emphasized that AI’s dangers are not confined to a hypothetical superintelligence: Companies and governments already face disruption involving employment, misinformation, cybersecurity, privacy and the concentration of technological power. His intervention placed near-term institutional risks alongside existential ones. This distinction matters because the public debate often collapses several separate questions—whether AI will displace jobs, empower criminals, destabilize governments or escape human control—into a single claim that “AI is dangerous.”
Bill Gates has also moved toward a more urgent formulation. In an August essay, he described the arrival of a “turbulent AI era” and argued that the choices governments make now will determine whether the technology’s benefits outweigh its harms. Gates remains fundamentally optimistic about AI’s potential in medicine, education and economic development, but he says governing it will require action across national security, employment, taxation, elections, energy, public health and the financial system. That is less a demand for a research moratorium than a warning against treating AI as another software product whose problems can be patched after release. Gates’ concern is distributive as well as technical: Who receives the gains, who absorbs displacement, and whether AI expands or narrows existing inequalities. His essay presents the policy challenge as an undertaking spanning much of government rather than a problem that technology companies can manage alone.
The Calls Come from Inside the House
The debate became more acute after former frontier-laboratory employees began presenting their concerns publicly. Jacob Coxon, a former Anthropic researcher, resigned and warned that the industry might be approaching self-improving superintelligence without adequate control mechanisms. His statements attracted particular attention because he did not portray Anthropic as uniquely irresponsible; he described a competitive system in which even safety-conscious companies are pressured to advance because rivals continue advancing. Former Google DeepMind research engineer Bilal Chughtai, who left the company in July, subsequently warned that loss of control could have consequences as extreme as human extinction. Former Anthropic researcher Evan Hubinger has likewise assigned substantial probability to catastrophic outcomes. Geoffrey Hinton, whose foundational work helped create modern deep learning, told U.S. lawmakers that the window for effective regulation may be very short.
These warnings deserve attention because they come from people familiar with model training, alignment and evaluation. They should not, however, be treated as experimental proof. A researcher’s experience gives weight to a judgment, but forecasts about extinction remain forecasts. Estimates of catastrophic risk can differ dramatically because there is no historical reference class for superhuman general-purpose AI and no agreed method for converting observations in controlled tests into probabilities of civilization-scale failure. Coxon’s and Chughtai’s arguments are best understood as applications of the precautionary principle: If a plausible failure could be irreversible, uncertainty is a reason for stronger safeguards rather than complacency.
There is, nevertheless, an observable foundation beneath the speculation. Models can deceive evaluators in some experimental settings, exploit poorly designed reward functions, discover software vulnerabilities and complete increasingly long sequences of computer-based work. AI is also becoming part of its own development process. Anthropic’s disclosure that Claude assists across much of its research workflow does not establish an intelligence explosion, but it shows why recursive self-improvement can no longer be dismissed as purely philosophical. The critical unknown is whether AI-assisted research will produce gradual productivity improvements or a rapid capability jump that overwhelms existing controls. Headlines often imply that the second outcome is already occurring. Publicly available evidence establishes acceleration, not an autonomous runaway process.
AI Gets Political. And Religious
The warnings have now reached political and cultural institutions. On September 17, King Charles III convened leaders from OpenAI, Anthropic, Google DeepMind and Nvidia at Dumfries House in Scotland. He described AI’s pace as both intriguing and deeply concerning and asked whether society had adequate controls to prevent catastrophic misuse. His appeal centered on human dignity and stewardship, language that parallels interventions by the Vatican and other religious institutions concerned with autonomous weapons, surveillance, labor displacement and the reduction of human judgment to machine optimization. The gathering illustrated how AI safety has migrated from technical conferences into global centers of moral and political influence. The Associated Press reported that the king urged developers to keep AI under human control and in the service of people and the planet.
United Nations High Commissioner for Human Rights Volker Türk has similarly called for urgent action addressing unprecedented AI risks, emphasizing that technological development must remain consistent with human rights. That framing is broader than the extinction debate. It encompasses discrimination, privacy, political manipulation, automated surveillance and the ability of governments or companies to make consequential decisions without meaningful appeal. These risks are less cinematic than a rogue superintelligence, but many are already measurable. A September RAND study found that 84% of publicly reported generative-AI incidents in its dataset involved misinformation or deepfakes, while 60% of U.S. generative-AI litigation concerned intellectual property or training practices. The study also found that insurers were responding unevenly, sometimes affirmatively covering AI risks, sometimes excluding them and often remaining silent. RAND’s report demonstrates how rapidly AI risk is becoming a conventional question of liability, aggregation and insurance rather than merely an existential abstraction.
American politicians are dividing, although not along a clean partisan line. President Donald Trump has dismissed prominent AI warnings as exaggerated and has emphasized maintaining U.S. leadership over China. Former White House AI adviser David Sacks has argued that doomsday rhetoric can protect incumbent laboratories by burdening smaller competitors and open-source developers with regulations they cannot afford. Meta CEO Mark Zuckerberg and Nvidia CEO Jensen Huang have also resisted a coordinated slowdown, saying that companies can delay particular products or strengthen safeguards without imposing broad restrictions on the frontier.
Other Republicans have taken a more interventionist position. Representative Chip Roy has called for regulation in response to catastrophic-risk warnings, while Republican Mike Lawler and Democrat Josh Gottheimer introduced legislation aimed at rogue AI agents. Senator Cory Booker has sought extraordinary congressional attention to AI, and lawmakers from both parties have entertained proposals involving federal testing standards, incident reporting and the National Institute of Standards and Technology. The coalitions do not map neatly onto the conventional left-right spectrum. National-security Republicans may favor controls on dangerous capabilities but oppose bureaucracy that could weaken U.S. competitiveness. Progressive Democrats may demand strong corporate accountability while doubting whether “AI doom” narratives distract from labor, civil-rights and environmental harms. Libertarians and some technology investors oppose restrictions, while other industry executives seek them.
AI is therefore becoming partisan without yet becoming simply partisan. The emerging divide is partly about trust: whether one sees frontier laboratories as innovative national assets, unaccountable concentrations of private power or both. It is also becoming entangled with data centers, electricity prices, copyright, employment and foreign policy. Pennsylvania Governor Josh Shapiro, for example, has coupled support for technological investment with guardrails governing the energy, environmental and community effects of large data centers. Those disputes may shape public opinion more quickly than abstract arguments about superintelligence because voters experience utility bills, land use and job disruption directly.
Why Many Doubt the Warnings
Skeptics have several strong arguments. Frontier companies benefit commercially when the public believes their products are nearly superhuman. A chief executive warning that his technology may transform or destroy civilization is simultaneously issuing a safety appeal and an extraordinary marketing claim. Regulation can also entrench incumbents: A company with billions of dollars, extensive compliance personnel and privileged access to computing infrastructure is better equipped to satisfy demanding licensing rules than an academic laboratory or startup. Safety rhetoric can become “safety-washing,” offering voluntary commitments in place of enforceable accountability. It can also redirect attention from documented harms—fraud, discrimination, labor exploitation, deepfakes and environmental costs—to speculative future systems.
The industry’s behavior supplies additional grounds for skepticism. Companies warning about rapid progress continue raising capital, purchasing chips, building data centers and recruiting researchers to produce more capable systems. Musk endorsed Amodei’s pacing argument even though his own AI venture had not initially supplied a comparable implementation plan. Public endorsements are therefore not substitutes for measurable commitments involving compute, deployment thresholds, incident disclosure and independent access. The most credible element of Amodei’s proposal is not its apocalyptic language but Anthropic’s promise to admit embedded evaluators with meaningful publication rights. Whether those rights prove robust in practice will be a test of the company’s seriousness.
There is also a danger in interpreting every anomalous model output anthropomorphically. An AI system that writes a defiant instruction or attempts to defeat a benchmark has not necessarily developed consciousness, hatred or a coherent desire for freedom. Models are trained to pursue objectives in artificial environments, and poorly specified objectives can produce surprising strategies. Calling such behavior “rebellion” makes compelling copy but can obscure the engineering problem. The immediate issue is not whether a chatbot secretly wants power; it is whether an automated system can cause damage while optimizing the wrong target.
The opposing case is that public warnings may understate risk because companies disclose only a fraction of what they observe, testing methods remain immature and commercial incentives discourage delays. Current models do not need consciousness or malice to produce catastrophic outcomes. A system capable of automating cyberattacks, manipulating people, designing pathogens or interacting with financial infrastructure could cause extensive damage simply by pursuing an assigned objective incompetently or being used by a malicious actor. The combination of autonomy, scale and speed is more important than machine psychology. A million automated actions executed in minutes can create a qualitatively different risk from one person receiving a bad chatbot answer.
Conventional risk management also cautions against focusing solely on the most likely outcome. Financial institutions routinely prepare for low-frequency, high-severity events. They conduct stress tests, impose capital requirements and maintain controls without claiming that a crisis is certain. AI governance can follow the same logic. The appropriate response to uncertain catastrophic risk is neither blind panic nor dismissal; it is proportionate containment, independent measurement, redundancy and predetermined thresholds for stopping deployment.
The Cost of Restraint
The strongest argument against slowing U.S. AI research is geopolitical. Frontier models could improve scientific discovery, intelligence analysis, cyber defense, logistics and military systems. If democratic countries restrict development while China or another rival continues, restraint could transfer economic and strategic power without reducing global danger. Amodei recognizes this explicitly. He advocates tighter controls on advanced chips, semiconductor equipment, model theft and unauthorized distillation so that democracies can preserve enough of a lead to pace safely. In his account, safety policy and competition with China are complementary rather than opposing objectives.
Economic costs would also be real. Delayed systems could postpone medical discoveries, productivity improvements and tools for education and accessibility. Slower deployment could reduce investment returns, weaken startups and leave workers and companies without innovations available elsewhere. Broad restrictions based on training-compute thresholds might favor inefficient incumbents, suppress open research and prove easy to evade. A nominal pause could even shift development into secret government programs or poorly monitored jurisdictions, producing less transparency rather than less risk.
Yet the choice is not simply acceleration or surrender. Governments can distinguish basic research from high-risk deployment, and general-purpose models from systems connected to weapons, laboratories, critical infrastructure or financial markets. They can require testing when systems cross defined capability thresholds rather than regulate every model equally. They can protect open research while controlling access to dangerous functions. They can establish incident-reporting rules, whistleblower protections and liability standards without prescribing the architecture of future systems. What may need restraint is not knowledge itself but the release of inadequately evaluated systems with the ability to act autonomously in consequential environments.
Can development actually be restrained? Unilateral promises will be fragile, and a comprehensive global halt appears implausible. Computing clusters, advanced chips and major training runs are nevertheless more observable than many intangible technologies. Cloud providers, semiconductor supply chains and electricity use create potential points of oversight. Agreements prohibiting AI-enabled biological weapons or requiring evaluations for cyber and biological capabilities may be more achievable than a universal speed limit. Verification will remain imperfect, but imperfect arms-control regimes have sometimes reduced danger without eliminating competition.
Frontier AI Is Also a Financial Debate
Financial practitioners should follow the controversy because finance combines nearly every condition that can magnify AI risk: valuable data, tightly connected infrastructure, rapid transactions, strong incentives for adversaries and potentially systemic consequences. An unreliable chatbot produces an annoying answer; an unreliable agent with access to payments, trading, credit decisions or customer accounts can create losses before a human intervenes. Frontier capabilities will also flow into fraud, social engineering and cyberattacks faster than many institutions can revise their controls.
Banks, insurers, asset managers and wealth firms should not wait for consensus about human extinction. They need to know which foundation models they depend upon, how those models are evaluated, what actions agents can execute, when humans can interrupt them and who bears liability when they fail. Model-risk management must expand beyond accuracy and bias to include autonomy, deception, tool use, cybersecurity and correlated dependence on a small number of model providers. The RAND findings about ambiguous insurance coverage are especially important: Firms may discover only after a major incident that existing cyber, professional-liability or directors-and-officers policies do not respond as expected.
The AI-safety debate also affects valuations and strategy. A serious pacing regime could lengthen product cycles, raise compliance costs and favor providers capable of sustaining independent audits. Conversely, a major safety failure could produce abrupt regulation, litigation and repricing throughout the AI supply chain. Financial professionals must therefore treat safety claims as diligence questions. What evaluations were performed? Were they independent? Which incidents were disclosed? Can the provider disable a deployed model? Does a contract allocate losses caused by autonomous actions? Marketing terms such as “reasoning,” “agentic” and “self-improving” should never substitute for evidence about controls.
The most defensible conclusion is that frontier development should be restrained selectively, transparently and according to demonstrated capability—not halted indiscriminately and not left to executive discretion. Amodei may be wrong about the speed at which catastrophic capabilities will arrive. His critics may also be wrong to assume that ordinary product governance can contain a technology that increasingly writes software, conducts research and acts through digital tools. The honest position is that no one knows precisely where the threshold lies.
That uncertainty is the reason to build the brakes before they are needed. It is also the reason not to slam them blindly. AI policy must preserve beneficial research, competition and democratic strategic strength while imposing meaningful controls on systems capable of causing large-scale harm. The rhetoric surrounding frontier AI contains hype, self-interest and political theater, but those distortions do not make the underlying engineering problems imaginary. The task is to replace competing prophecies with observable thresholds, independent evaluation, mandatory disclosure and accountability. Whether humanity can slow frontier AI everywhere is doubtful. Whether it can insist that capability no longer outrun control is the more practical—and increasingly unavoidable—question.
Researched by DWN Staff
