Chinese researchers outline AI that can build its successors

Thirty-five Chinese AI researchers, including authors affiliated with ByteDance, Tsinghua and Shanghai AI Lab, have published “The Last AI Built by Humans”, a roadmap for AI that can take charge of its own improvement and build its successors.

As The Rundown reported, the paper sets out five levels of recursive self-improvement, or RSI. It first appeared on September 10 and was revised on September 15.

Five levels of AI improvement

Level 1 has AI carrying out upgrades humans designed. Level 2 lets it diagnose its weak spots and decide how to fix them.

Levels 3 and 4 hand over decisions about what the model learns next and how it adapts after deployment. At Level 5, AI can redesign the improvement process itself.

The authors classified 491 existing papers on this ladder, with 75.4% at Levels 1 and 2 and 5.9% at Level 5. Those figures describe the authors’ classification of research. Sustained gains across successive model generations remain an open problem.

The researchers see coding as the clearest route to RSI because software changes can get immediate feedback from tests. Robotics, science and medicine face slower, more costly feedback. The tests still need to establish that a change improved the system.

Why it matters

The researchers present RSI as a milestone. OpenAI, Anthropic and Google all treat substantial automation of AI research as a risk area because it could push new capabilities beyond effective oversight.

OpenAI’s Preparedness Framework places fully automated AI research and development at its Critical threshold. One indicator is sustained improvements across model generations taking one-fifth as long as equivalent work in 2024. The company warns that such acceleration could overwhelm oversight. At that threshold, its policy specifies halting further development until appropriate safeguards and security controls have been defined.

Anthropic’s Responsible Scaling Policy effective July 8 sets a threshold around replacing its entire research scientist and research engineer workforce at competitive costs, or sharply accelerating development through AI research automation. The company plans security research and comprehensive logging of internal AI development, while acknowledging that the threshold has been hard to apply.

Google DeepMind’s Frontier Safety Framework sets a threshold at fully automating a Google team focused on improving AI capabilities, at roughly comparable total costs. It warns that stronger successors could emerge faster than effective defenses. It also considers risks from deploying systems internally.

For labs and safety teams, faster research could shorten the time available to review each improvement. An agent working only inside a lab could shape later models before any public release. That puts controls over research infrastructure, records of changes and authority to approve new models at the center of the oversight problem.

The national comparison has limits. The paper reflects its authors’ outlook. AP reported that Guo Jiakun, a spokesperson for China’s foreign ministry, rejected AI warnings aimed at China while calling for cooperation on global AI governance. That response suggests resistance to the geopolitical framing of the debate. China’s broader tolerance for RSI risk remains unclear.

The roadmap allows people to set objectives, protect evaluations, grant resource access and veto releases even at Level 5. Its ladder measures control over the improvement process, while the lab frameworks assess the speed, scale and risks of research automation. A Level 5 label alone therefore gives an incomplete picture of the safety question.

Future RSI claims will need to show how much research a system completes, how long it takes, what it costs and how much human help it needs. Labs evaluating those claims will also need clear answers about which changes the system can make, who checks them and who can block a release.