Anthropic and OpenAI leaders join warning about AI that improves itself

Anthropic’s Jack Clark and OpenAI’s Jakub Pachocki have joined Geoffrey Hinton, Yoshua Bengio and other researchers in a paper urging preparations for an “intelligence explosion.” It explores how AI systems doing research could build better versions of themselves and compress years of advances into months.

The warning, covered in The Rundown, was published Sept. 28. The authors propose limits on how fast AI capabilities advance, outside auditors working inside labs and ways to stop risky research experiments.

How research could accelerate

Anthropic reported that AI completed 26% of its internal AI research and development work in August 2026 under overall human supervision, up from 1% in March, according to GovAI’s summary.

One calculation in the paper assumes full automation of AI research and development, continued gains from research at the estimated rate and no additional bottlenecks. Under those conditions, the authors estimate that progress could accelerate roughly tenfold within 1.5 years of full automation. A year of today’s AI progress would then take about five weeks.

The paper identifies uncertainty about further gains from research and possible limits on computing resources. It presents “the marginalization or extinction of humanity” as an extreme possible consequence of losing control.

Why it matters

Hinton and Bengio have become familiar voices warning about advanced AI. The more striking development is Clark and Pachocki joining them. Their participation points to a growing shared concern inside competing labs, although the paper says its authors’ views do not necessarily represent their organizations.

Safety talks give that convergence more substance. Bloomberg reported on Sept. 15 that OpenAI policy chief Chris Lehane said discussions with Anthropic and Google DeepMind had been underway for several weeks. Whether those discussions will produce enforceable limits remains uncertain.

Anthropic’s Dario Amodei committed on Sept. 12 to bringing in evaluators with substantial internal access and rights to publish findings, subject to specified redactions. TechCrunch reported that OpenAI’s Sam Altman committed to matching evaluator access.

For researchers inside labs, embedded oversight could bring scrutiny to training and internal research practices before a model reaches users. The value of those commitments depends on when evaluators begin work, what they can inspect and how redaction rules affect their ability to publish troubling findings.

Amodei describes pacing as compatible with continued model training and technical progress. Limits on capability growth or a halt to a risky experiment could give evaluators more time to assess dangers. Those controls could also delay research. The paper allows that AI could accelerate safety work, so restrictions could carry costs for that work too.

For policymakers, the paper’s proposed reporting on automation and progress could help show when extra checks are warranted. Anthropic’s figure illustrates the kind of information involved. Making it useful for oversight would require measurements that outsiders can verify and compare, with a clear account of human supervision.

The U.S. and China agreed Sept. 26 to establish an AI incident communication channel and hold a dialogue in November. Those plans could improve crisis communication. Trump nevertheless rejected slowing U.S. AI efforts, leaving a gap between Washington’s position and the calls for restraint emerging from inside AI labs.