The word “recursion” is the latest buzzword in AI circles. Two separate startups have taken on the name, and many more have started referencing recursive self-improvement (RSI) in their roadmaps. Like AGI before it, RSI has become a three-letter byword for a cataclysmic AI takeoff – even if there’s still a little disagreement about what it exactly means.
In basic terms, RSI refers to an AI system that can continuously upgrade itself. Once AI systems can manage the upgrade cycle better than humans, the process can become a closed loop, limited only by the compute power they can access, and humans are no longer necessary or even helpful.
Scary or not, that’s a vision that a lot of AI labs are eager to chase.
Earlier this month, well-known AI researcher Richard Socher launched the aptly named Recursive Superintelligence with RSI as an explicit goal. “Our main focus is to build truly recursive, self-improving superintelligence at scale,” Socher told TechCrunch at launch, “which means that the entire process of ideation, implementation, and validation of research ideas would be automatic.”
A number of other prominent researchers are already chasing that same goal, hoping for a breakthrough that will make recursive self-improvement possible.
One of the most prominent is Andrej Karpathy, a legendary figure from Tesla and OpenAI, who is using agent swarms to train LLMs on simple tasks for a project he calls Auto-Research. Karpathy has been unusually open about the project, tweeting about milestones regularly and making the building blocks available through a public GitHub repo. So far, the work has mostly been confined to making minor improvements on a GPT-2 scale model — as Karpathy noted in March, “It’s not novel, ground-breaking ‘research’ (yet)” — but it’s been enough to convince lots of other researchers to follow the RSI dream. And with Karpathy now working on pre-training at Anthropic, he will have plenty of opportunity to apply the idea at a larger scale.
Adaption — founded by Cohere and Google alum Sara Hooker — recently launched a similar tool called AutoScientist in an effort to automate frontier training. Like Karpathy’s auto-researchers, the system trains agents to make incremental improvements — but for Adaption, the goal is to make it easier to train a full-scale frontier model. If those same researchers start to push the frontier forward, the system could quickly spiral into something very much like RSI.
Disarray founder Doris Xin drew more specific RSI interest when her self-trained machine learning agent took home 28 medals in a recent Kaggle competition, beating out many human-trained agents. As she sees it, the major challenge is reliability.
“I would argue, given infinite compute and infinite time horizon, we are already there,” Xin told me. “I want to make an argument that this is not a creative endeavor, really. It’s just a lot of meat-and-potatoes engineering.”
The word “recursion” is the latest buzzword in AI circles. Two separate startups have taken on the name, and many more have started referencing recursive self-improvement (RSI) in their roadmaps. Like AGI before it, RSI has become a three-letter byword for a cataclysmic AI takeoff – even if there’s still a little disagreement about what it exactly means.
In basic terms, RSI refers to an AI system that can continuously upgrade itself. Once AI systems can manage the upgrade cycle better than humans, the process can become a closed loop, limited only by the compute power they can access, and humans are no longer necessary or even helpful.
Scary or not, that’s a vision that a lot of AI labs are eager to chase.
Earlier this month, well-known AI researcher Richard Socher launched the aptly named Recursive Superintelligence with RSI as an explicit goal. “Our main focus is to build truly recursive, self-improving superintelligence at scale,” Socher told TechCrunch at launch, “which means that the entire process of ideation, implementation, and validation of research ideas would be automatic.”
Not there yet
There’s also plenty of evidence that the AI industry isn’t very close to recursive systems in any meaningful way — and is still grappling with talking to a wary public about its progress. So Google CEO Sundar Pichai basically admitted in a recent podcast interview.
“It’s a continuum, and we are all definitely making progress,” Pichai said. “But in the way people describe RSI, that would represent a next level of acceleration and would have a lot of implications, but we aren’t quite there yet.”
But the continuum includes an awful lot of self-improving AI systems. In January, one of Anthropic’s lead programmers for Claude Code estimated that “close to 100%” of his team’s code was written by the tool — a frank admission that Claude Code was literally writing itself.
Just because engineers are using an AI tool doesn’t mean the tool can replace them — but Anthropic seems to be getting close to replacing engineers too. In a recent survey tied to the Mythos preview, five out of 18 Anthropic engineers believed that, with harness improvements, this version of Mythos could soon substitute for an L4 engineer — a midlevel programmer who can take on involved projects without supervision.
Still, there were some of the same weaknesses you might expect.
“Some of Claude’s major reported weaknesses compared to an L4 include: self-managing week-long ambiguous tasks, understanding org priorities, taste, verification, instruction-following, and epistemics,” the report reads.
In other words, its weaknesses are everything involved with self-direction, which is the cornerstone for RSI. But sure, for everything else, Claude is ready to step right in.
Just like the AGI term before it, the AI industry also can’t tell us how far away it is from showcasing a meaningful recursive system. When Georgetown’s Center for Security and Emerging Technology assembled a group of experts to study RSI last year, the group found a major split in assessments — some expecting an imminent “superintelligence” style explosion while others expected slower progress and an eventual plateau. But all agreed that recursion made the future especially difficult to predict.
Helen Toner, director of CSET and a former board member at OpenAI, told TechCrunch that simply using AI tools to do AI research isn’t enough to qualify as RSI. “They’re just using AI for as much as they can,” Toner told TechCrunch. “And I think that is different from the classic definition of RSI, which is really that there are no humans needed.”
Toner pointed to a recent post by METR’s Ajeya Cotra, which distinguishes different milestones on the path to the AI research takeover. One step, which Cotra calls “adequacy,” would come when the system can still perform research after all humans are removed — even if the resulting research isn’t as valuable or efficient. “Parity” comes when an AI-only system is as good at research as a human-only system. “Supremacy,” the final stage, comes when an AI-only system outperforms a collaborative system between humans and AI.
Ultimately, Cotra concludes that AI is very close to the adequacy threshold of being able to produce some work on its own — similar to the incremental changes made by Karpathy’s Auto-Research system. “I wouldn’t be totally shocked if you told me this milestone had already passed, and I expect it to happen in the next couple years,” Cotra wrote.
She was less clear on when parity will come, but once it does, she thinks it would “massively accelerate the pace of AI progress, leading to AI research supremacy within another year.”
Bumps in the road
With so much of AI built on scaling laws, there’s a strong tendency to think RSI will follow the same curve. Toner thinks that many of those pursuing AI research and development via RSI “think of it as a pretty smooth ladder, where you can just keep scaling up.”
But even if AI researchers are able to make incremental improvements like Karpathy’s auto-researchers, there will be larger challenges in handing off the whole process of research. Toner put it in terms of the history of computing, which has seen human beings handing off more and more of the process while still directing things from the top.
“We went from machine languages to assembly language and compiled languages; you’re getting further and further from the guts of the computer,” Toner said. “But the human is still, in some intuitive sense, running the show.”
Moving beyond that paradigm will take significant challenges, both in engineering and alignment. But even with the massive investments happening, there’s no infinite compute available — and the basic trade-off between human labor and machine intelligence will be hard to overcome.
As for a total recursive AI system of apocalyptic visions? The only thing researchers essentially agree on is that, like AGI, it’s not here yet.
Topics
A number of other prominent researchers are already chasing that same goal, hoping for a breakthrough that will make recursive self-improvement possible.
One of the most prominent is Andrej Karpathy, a legendary figure from Tesla and OpenAI, who is using agent swarms to train LLMs on simple tasks for a project he calls Auto-Research. Karpathy has been unusually open about the project, tweeting about milestones regularly and making the building blocks available through a public GitHub repo. So far, the work has mostly been confined to making minor improvements on a GPT-2 scale model — as Karpathy noted in March, “It’s not novel, ground-breaking ‘research’ (yet)” — but it’s been enough to convince lots of other researchers to follow the RSI dream. And with Karpathy now working on pre-training at Anthropic, he will have plenty of opportunity to apply the idea at a larger scale.
Adaption — founded by Cohere and Google alum Sara Hooker — recently launched a similar tool called AutoScientist in an effort to automate frontier training. Like Karpathy’s auto-researchers, the system trains agents to make incremental improvements — but for Adaption, the goal is to make it easier to train a full-scale frontier model. If those same researchers start to push the frontier forward, the system could quickly spiral into something very much like RSI.
Disarray founder Doris Xin drew more specific RSI interest when her self-trained machine learning agent took home 28 medals in a recent Kaggle competition, beating out many human-trained agents. As she sees it, the major challenge is reliability.
“I would argue, given infinite compute and infinite time horizon, we are already there,” Xin told me. “I want to make an argument that this is not a creative endeavor, really. It’s just a lot of meat-and-potatoes engineering.”
Not there yet
There’s also plenty of evidence that the AI industry isn’t very close to recursive systems in any meaningful way — and is still grappling with talking to a wary public about its progress. So Google CEO Sundar Pichai basically admitted in a recent podcast interview.
“It’s a continuum, and we are all definitely making progress,” Pichai said. “But in the way people describe RSI, that would represent a next level of acceleration and would have a lot of implications, but we aren’t quite there yet.”
But the continuum includes an awful lot of self-improving AI systems. In January, one of Anthropic’s lead programmers for Claude Code estimated that “close to 100%” of his team’s code was written by the tool — a frank admission that Claude Code was literally writing itself.
Just because engineers are using an AI tool doesn’t mean the tool can replace them — but Anthropic seems to be getting close to replacing engineers too. In a recent survey tied to the Mythos preview, five out of 18 Anthropic engineers believed that, with harness improvements, this version of Mythos could soon substitute for an L4 engineer — a midlevel programmer who can take on involved projects without supervision.
Still, there were some of the same weaknesses you might expect.
“Some of Claude’s major reported weaknesses compared to an L4 include: self-managing week-long ambiguous tasks, understanding org priorities, taste, verification, instruction-following, and epistemics,” the report reads.
In other words, its weaknesses are everything involved with self-direction, which is the cornerstone for RSI. But sure, for everything else, Claude is ready to step right in.
Just like the AGI term before it, the AI industry also can’t tell us how far away it is from showcasing a meaningful recursive system. When Georgetown’s Center for Security and Emerging Technology assembled a group of experts to study RSI last year, the group found a major split in assessments — some expecting an imminent “superintelligence” style explosion while others expected slower progress and an eventual plateau. But all agreed that recursion made the future especially difficult to predict.
Helen Toner, director of CSET and a former board member at OpenAI, told TechCrunch that simply using AI tools to do AI research isn’t enough to qualify as RSI. “They’re just using AI for as much as they can,” Toner told TechCrunch. “And I think that is different from the classic definition of RSI, which is really that there are no humans needed.”
Toner pointed to a recent post by METR’s Ajeya Cotra, which distinguishes different milestones on the path to the AI research takeover. One step, which Cotra calls “adequacy,” would come when the system can still perform research after all humans are removed — even if the resulting research isn’t as valuable or efficient. “Parity” comes when an AI-only system is as good at research as a human-only system. “Supremacy,” the final stage, comes when an AI-only system outperforms a collaborative system between humans and AI.
Ultimately, Cotra concludes that AI is very close to the adequacy threshold of being able to produce some work on its own — similar to the incremental changes made by Karpathy’s Auto-Research system. “I wouldn’t be totally shocked if you told me this milestone had already passed, and I expect it to happen in the next couple years,” Cotra wrote.
She was less clear on when parity will come, but once it does, she thinks it would “massively accelerate the pace of AI progress, leading to AI research supremacy within another year.”
Bumps in the road
With so much of AI built on scaling laws, there’s a strong tendency to think RSI will follow the same curve. Toner thinks that many of those pursuing AI research and development via RSI “think of it as a pretty smooth ladder, where you can just keep scaling up.”
But even if AI researchers are able to make incremental improvements like Karpathy’s auto-researchers, there will be larger challenges in handing off the whole process of research. Toner put it in terms of the history of computing, which has seen human beings handing off more and more of the process while still directing things from the top.
“We went from machine languages to assembly language and compiled languages; you’re getting further and further from the guts of the computer,” Toner said. “But the human is still, in some intuitive sense, running the show.”
Moving beyond that paradigm will take significant challenges, both in engineering and alignment. But even with the massive investments happening, there’s no infinite compute available — and the basic trade-off between human labor and machine intelligence will be hard to overcome.
As for a total recursive AI system of apocalyptic visions? The only thing researchers essentially agree on is that, like AGI, it’s not here yet.