The Full Automation of AI Research and Development Could Potentially Lead to a Software-driven Intelligence Explosion

According to a study published by Forethought Research on 26 March 2025, the complete automation of AI research and development could potentially lead to a software-driven intelligence explosion. The researchers examined what happens when AI systems become capable of fully automating their own development processes, creating a feedback loop where each new system generates even more advanced AI, potentially within months.

Empirical data indicate that the efficiency of AI software roughly doubles every six months, and this progress is likely outpacing the growth rate of research resources. Analyses suggest that AI tools currently generate over 25% of Google’s entire source code automatically. At the same time, at Amazon, they have saved approximately 4,500 developer-years of work and an estimated $260 million in annual efficiency gains. The study identified two main obstacles to the full automation of AI systems for AI R&D (AI Systems for AI R&D Automation, ASARA) and a software-driven intelligence explosion (Software Intelligence Explosion, SIE): fixed computational capacity and the lengthy training times for new AI systems. However, the study argues that these limitations are unlikely to entirely prevent the emergence of an SIE, as algorithmic advancements have consistently improved training efficiency in the past—AI experiments and training processes may gradually become faster, enabling continued acceleration despite these barriers.

The pace of technological advancement could easily outstrip society’s ability to prepare, prompting the study to propose specific safety measures to manage the risks of a software-driven intelligence explosion. An SIE could lead to an extremely rapid evolution of current AI capabilities, necessitating the prior development of appropriate regulations. Proposed measures include continuous monitoring of software development, preemptive evaluation of AI systems’ development capabilities, and establishing a threshold that companies commit not to exceed without adequate safety measures, thereby ensuring the safe progression of technological advancement.

Source:

1.

Will AI R&D Automation Cause a Software Intelligence Explosion? | Forethought

AI companies are increasingly using AI systems to accelerate AI research and development. Today’s AI systems help researchers write code, analyze research papers, and generate training data. Future systems could be significantly more capable – potentially automating the entire AI development cycle from formulating research questions and designing experiments to implementing, testing, and refining new AI systems. We argue that such systems could trigger a runaway feedback loop in which they quickly develop more advanced AI, which itself speeds up the development of even more advanced AI, resulting in extremely fast AI progress, even without the need for additional computer chips. Empirical evidence on the rate at which AI research efforts improve AI algorithms suggests that this positive feedback loop could overcome diminishing returns to continued AI research efforts. We evaluate two additional bottlenecks to rapid progress: training AI systems from scratch takes months, and improving AI algorithms often requires computationally expensive experiments. However, we find that there are possible workarounds that could enable a runaway feedback loop nonetheless.

Forethought

2.

How AI Can Automate AI Research and Development

Technology companies are using AI itself to accelerate research and development for the next generation of AI models, a trend that could lead to runaway technological progress. Policymakers and the public should be paying close attention to AI R&D automation to prepare for how AI could transform the future.

RAND CorporationGaurav Sett

3.

Will AI R&D Automation Cause a Software Intelligence Explosion? — AI Alignment Forum

Empirical evidence suggests that, if AI automates AI research, feedback loops could overcome diminishing returns, significantly accelerating AI progr…

Tom Davidson