The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement

Evan: Welcome to Daily Paper Cast.

Ashley: Today's paper is from the Hugging Face daily paper list of September 16, 2026, and it has garnered 83 upvotes.

Evan: The paper we're discussing is titled 'The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement'.

Ashley: The first two authors are Yi Duan and Ying Liu, and the corresponding author is Xuanhe Zhou from Shanghai Jiao Tong University.

Evan: Let's dive into the Introduction section to understand what this paper is all about.

Ashley: Recent advancements in frontier-model development showcase several forms of scaling in the improvement pipeline.

Models like Kimi K3 and Qwen3.8-Max, containing over 2 trillion parameters each, illustrate the expansion.

These models support a context window of about one million tokens, demonstrating the monumental scale and intricacy involved.

Evan: That's a substantial scale.

But with such large models, what challenges do developers face during their development?

Ashley: Exactly.

Despite the growing use of agentic tools and API-based automation, developers shoulder the responsibility of determining what to improve, constructing the necessary resources, and validating the impact of changes.

As AI systems tackle increasingly demanding tasks, the comprehensive scale of development processes becomes a significant bottleneck.

Evan: So, what specific challenges arise at different stages of the model lifecycle?

Ashley: Three primary challenges emerge.

Firstly, foundation-model training is inherently resource-intensive, encompassing everything from data preparation to distributed optimization and evaluation.

For instance, OpenAI's latest advancements report hundreds of experiments on speculative-decoding draft models, leading to significant improvements but requiring extensive resources.

Ashley: Secondly, scaling feedback and learning environments are vital yet complex.

Synthetic data and reinforcement learning automate capabilities development but also demand sophisticated experience-generation infrastructure and reliable evaluation methods.

Lastly, recurring adaptation after deployment is crucial.

Deployed systems must handle evolving documents, new tools, incomplete contexts, and interdependent actions.

Continuous improvements necessitate manual diagnosis, interface management, and repeated regression testing.

Evan: Given these challenges, how does the concept of Recursive Self-Improvement, or RSI, come into play?

Ashley: RSI addresses these barriers by embedding part of the coordination into the AI system itself.

The core idea is that an AI system can autonomously identify its limitations, develop and validate improvements, and use these enhanced capabilities to refine its own improvement processes.

It's an autonomous, closed-loop process aimed at achieving continuous enhancement.

Evan: That sounds transformative.

How is RSI structured to achieve such autonomy, efficiency, and innovation?

Ashley: RSI is structured around three dimensions: autonomy, efficiency, and innovation.

Autonomy broadens the system's responsibility from executing predefined updates to identifying weaknesses, proposing changes, and retaining successful transformations.

Efficiency leverages resources such as data, compute, and human review to derive more validated improvements.

Meanwhile, innovation allows the system to go beyond human-prescribed updates and incorporate beneficial discoveries into subsequent improvement rounds.

Evan: Could you give an example to illustrate the distinction between conventional improvements and RSI?

Ashley: Certainly.

Consider the case of foundation-model training.

Traditionally, experiments select better checkpoints but leave the selection process unchanged.

In contrast, an RSI system like A-Evolve-Training consolidates post-training outcomes into a persistent research policy, which then guides future experiments.

This change not only improves current development but also shapes how subsequent improvements are pursued.

Evan: That's insightful.

How about another scenario, perhaps in software engineering?

Ashley: In software engineering, a conventional approach might fix a bug but leave the recurring failures of the coding agent unaddressed.

An RSI approach, like Ouroboros, reviews deployment evidence to suggest changes to the agent's tools and implementation.

It tests and reviews revisions, replacing the runtime with accepted versions that persistently improve subsequent coding behaviors and diagnostic processes.

Evan: So, RSI not only addresses immediate challenges but also fundamentally alters how AI systems evolve by refining their own improvement mechanisms.

Ashley: Precisely.

While this covers the core background and objectives, the paper also details the main contributions of RSI research, which we'll explore next.

That’s the end of the Introduction section.

Evan: Let's move on to the methods proposed in the paper.

Ashley: The core of the paper introduces a five-level autonomy framework to categorize the spectrum of recursive self-improvement.

The levels are designated as B0 to L5, each representing a step in the complexity and autonomy of the improvement process.

Evan: Can you describe the characteristics of these levels?

Ashley: Of course.

B0 is the base level where improvement is confined to the current task.

The AI system may revise outputs iteratively but does not retain these changes as persistent improvement.

L1, labeled as Improvement Execution Autonomy, involves the AI executing human-defined improvement procedures, with accepted results retaining for future tasks.

L2, Improvement Strategy Autonomy, involves the AI diagnosing weaknesses using evaluation feedback and autonomously deciding how to improve the system.

Ashley: Moving to L3, Learning-Signal or Experience-Acquisition Autonomy, the AI system determines what experience to acquire next based on current state assessments, whereas L4, Environment Adaptation Autonomy, involves the AI using deployment interaction to revise persistent system state under external acceptance and governance rules.

Finally, L5, Recursive Inheritance Autonomy, is the pinnacle level where the system persistently revises mechanisms governing subsequent improvements, such as an improver or verifier.

Evan: That's quite a structured framework.

Can you give examples at each level to illustrate how they differ?

Ashley: Certainly.

At B0, consider a coding agent revising code using a human-provided test suite.

It remains at B0 if the fixes are confined to the current task.

At L1, imagine Google’s high-fidelity label curation pipeline.

Developers specify decision criteria, and an LLM applies these criteria to label candidate data, with humans then refining and verifying the labels.

In L2, an AI like Dropbox’s GEPA rewrites prompts to judge file relevance using performance-driven feedback.

Ashley: L3 can be illustrated by SEAgent, which converts trajectory feedback into a software guidebook that conditions later tasks.

At L4, PANDO admits or demotes reusable rules during deployment based on long-running interactions.

For L5, A-Evolve-Training revises its research policy based on gathered development scores and subsequent experiment outcomes, guiding future rounds of improvement autonomously.

Evan: That's a comprehensive set of examples.

What kind of datasets and metrics are used to evaluate these levels of RSI autonomy?

Ashley: For evaluation, diverse datasets encompassing scientific discovery, embodied intelligence, software engineering, and healthcare scenarios are utilized.

Metrics include task performance, improvement-loop efficiency, and reliability of the changes retained across rounds.

For instance, in SEAgent, the guidebook is evaluated based on its effectiveness in improving subsequent agent tasks in software environments.

Evan: How does the specific evaluation protocol differ across these application domains?

Ashley: Evaluation in science involves assessing experimental agents and hypothesis generation tools for their ability to improve subsequent research.

In embodied intelligence, evaluation metrics might include the robustness of skills acquired through interaction with physical environments and their transferability.

Software engineering primarily uses repository-grounded feedback and versioned inheritance to measure the effectiveness of coding agents.

Healthcare focuses on longitudinal feedback, patient outcomes, and expert reviews to validate the safe inheritance of clinical updates.

Evan: What mechanisms or techniques are essential to ensuring the reliability of these recursive self-improvement systems?

Ashley: Ensure reliability through mechanisms like selective freezing of components, targeted ablations, rollback protocols, and independent validation.

For example, Google's research on debugging NCCL watchdog timeouts distills debugging experience into structured workflows executed by an AI agent, maintaining reliability through controlled trials and automated checks.

Additionally, systems such as HyperAgents focus on meta-agent self-modification, employing archives to track improvements and identify patterns in successful agent generations.

Ashley: Moreover, verification protocols like those used in RQGM freeze evaluators within epochs and validate replacements against independent ground-truth anchors, safeguarding against evaluator exploitation.

Approaches like A-Evolve-Training set the stage for structural L5 by proving inheritance and analyzing effectiveness across research policies, worker contributions, and objective alignment.

Evan: So, the methods employed range from executing predefined improvements to progressively improving the very mechanisms behind AI enhancements, spanning various application domains.

That's the crux of the Methods section.

Ashley: Exactly.

Each level of RSI autonomy plays a crucial role, leveraging distinct mechanisms and evaluation protocols to ensure robust and efficient self-improvement.

This wraps up our discussion of the Method section.

Evan: Let's move on to the experiments and results section to see how these methods were evaluated.

Ashley: Sure.

The paper's experiments are designed to validate the different levels of RSI autonomy across various application domains — science, embodied intelligence, software engineering, and healthcare.

Evan: How did they set up the experiments for each of these domains?

Ashley: For the science domain, they used an AI4Sci system called HypoForge that distills scientific experience into procedural skills for hypothesis generation and testing.

They evaluated this system on its ability to produce validated scientific hypotheses and experimental designs that are reused across research tasks.

Evan: What metrics did they use to measure success in this domain?

Ashley: They focused on task performance, the accuracy of hypotheses, and the reuse rate of generated skills in new scientific tasks.

Improved performance on held-out tasks and reliability of tool usage were key indicators.

Evan: How about the embodied intelligence domain?

Ashley: In embodied intelligence, the system used was Voyager, which writes successful behaviors into a skill library.

They tested this in Minecraft, observing how well the agent attained new tasks by reusing skills.

Evan: And what were the key results from this experiment?

Ashley: The results showed that agents leveraging reusable skills had significantly improved performance in new tasks without manual intervention.

The effectiveness of skill reuse was evident in both the speed and accuracy of task completion.

Evan: Let's discuss the software engineering domain next.

What approach did they take there?

Ashley: They used the Self-Harness system, which proposed and tested changes to the coding agent’s harness.

By evaluating these proposed changes on hidden tasks and observing their persistence across multiple rounds, they assessed continuous improvement.

Evan: What specific metrics did they focus on?

Ashley: Metrics included task completion rate, code quality improvements, and the reduction of recurring errors across multiple task executions.

Key performance improvements were recorded and compared to initial capabilities.

Evan: Finally, how did they approach the healthcare domain experiments?

Ashley: They utilized EvoClinician in simulated diagnostic scenarios, where the agent's reasoning strategies were continuously updated based on feedback.

Improvements were retained and reused in subsequent cases.

Evan: What were the main findings from these healthcare tests?

Ashley: The findings showed that persistently updated diagnostic policies led to more accurate diagnoses over time.

The AI agent’s ability to better integrate and apply clinical feedback was evidenced by improved patient outcomes in simulated environments.

Evan: So, across these domains, the experiments validated that recursive self-improvement systems can indeed foster continuous enhancement.

Ashley: Exactly.

Each domain demonstrated how RSI contributes to progressive development, whether it's through validated scientific hypotheses, reusable skills in embodied intelligence, improved coding practices in software engineering, or accurate clinical reasoning in healthcare.

Evan: That concludes the experiments and results section.

Evan: Now that we've discussed the experiments and results, let's look at the related work referenced in the paper.

Ashley: Sure.

The related work section covers a comprehensive overview of existing research on self-evolving systems, classification frameworks, automated machine learning, and continual learning, among others.

The authors aim to position their work within the broader context of these fields.

Evan: How do they categorize self-evolving systems?

Ashley: They start by examining existing surveys that provide complementary taxonomies of self-evolving systems.

These surveys explore stages of self-evolution, specific objects of updates, timing, and technical mechanisms.

These views illuminate what changes within a system and how these changes occur, but they often overlook the autonomy and decision-making transferred to AI during these processes.

Evan: What do they propose differently in their classification?

Ashley: The authors propose the improvement loop as the central unit of analysis.

Instead of focusing solely on the stages or types of changes, they suggest examining what triggers an improvement, the entities proposing and validating changes, what persists, and which later decisions utilize the retained changes.

This creates a more holistic understanding of recursive self-improvement processes.

Evan: It's a shift from object-based categorization to process and decision-based analysis.

Ashley: Exactly.

They also operationalize autonomy by examining the improvement decisions transferred from external designers to AI, rather than focusing solely on model capability or the number of automated components.

This framework introduces five distinct levels, from executing predefined improvements to developing mechanisms governing future improvements.

It's designed to highlight which parts of an improvement process current systems can assume and what supports claims of recursive progress.

Evan: How does their work differentiate from AI for AI and continual learning paradigms?

Ashley: They differentiate RSI from neighboring paradigms like continual learning and automated machine learning (AutoML) by focusing on recursive self-improvement processes rather than single-stage optimization or incremental learning from data.

Continual learning often centers on a model acquiring knowledge from sequential tasks while limiting the loss of previous capabilities, but it typically doesn’t revise how learning rules or objectives are proposed and evaluated.

Ashley: AutoML automates decisions like model selection and hyperparameter tuning but generally operates within externally supplied objectives and constraints.

While these systems automate substantial parts of model development, their search spaces and evaluators are usually predefined.

Evan: So, RSI aims to improve not only task performance but also how later adaptations are discovered and integrated?

Ashley: Precisely.

RSI is about the system autonomously transforming acquired experience and feedback into persistent self-improvements, such that these changes can further affect the mechanisms generating, evaluating, and selecting subsequent improvements.

It's a step beyond continual learning and AutoML toward self-directed and self-improving AI systems.

Evan: Do they also discuss related research on aligning agent behavior and judging the performance of self-improving systems?

Ashley: Yes, they examine evaluation surveys that study model-based judgment, agent assessment, and rubric-guided learning.

These frameworks focus on ensuring higher task performance, but they stress that higher performance alone doesn’t prove revised mechanisms were retained and reused.

It's crucial to distinguish between structural recursion, where a revised mechanism governs a later round, and effective recursion, where the mechanism produces stronger successors under comparable budgets.

Evan: Do they talk about the importance of transparent protocols and verification efforts within RSI practice?

Ashley: Indeed.

Verification methods like execution traces, structured recurring tests, independent anchors for evaluator replacement, and audit logs are emphasized for validating recursive progress.

These structured verifications ensure that reported improvements stem from genuine gains in system capability rather than exploitation of evaluators or improvement protocols.

Evan: Therefore, the related work section not only situates their research within the existing landscape but also differentiates RSI by emphasizing process-based autonomy, recursive mechanism evolution, and rigorous verification protocols.

Ashley: This comprehensive approach provides a clearer picture of how iterative self-improvement can become a transformative process for AI systems.

That's the end of the Related Work section.

Evan: Alright, with the related work covered, let's wrap up by summarizing the key contributions and takeaways from the paper.

Ashley, where do we begin?

Ashley: Certainly.

The paper’s main contribution is the introduction of a comprehensive autonomy framework for recursive self-improvement, ranging from B0 to L5.

This framework provides a structured approach to categorize AI systems based on their level of self-improvement autonomy.

Evan: That framework indeed stands out.

Anything else?

Ashley: Yes, the paper also applies this framework across multiple domains: scientific research, embodied intelligence, software engineering, and healthcare.

It demonstrates how these levels of autonomy can be tailored to various applications, with concrete examples and detailed evaluations.

Evan: And how does this research advance the field?

Ashley: The authors argue that RSI goes beyond traditional AI development by not only improving task performance but also refining the very mechanisms behind AI enhancements.

This recursive approach ensures that the system continuously evolves, becoming increasingly efficient and innovative over time.

Evan: What about the practical implications of their findings?

Ashley: Practically, it suggests designing AI systems with mechanisms that can self-improve autonomously, making them more adaptive and capable of handling increasingly complex tasks with minimal human intervention.

This could significantly reduce the resource and labor-intensive processes traditionally associated with AI development.

Evan: In essence, the paper provides both a theoretical foundation and empirical validation for recursive self-improvement systems, positioning them as a pivotal development in advancing AI capabilities.

Ashley: And that brings us to the end of our discussion on 'The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement.'

Evan: We hope you found today's episode insightful.

Be sure to join us next time for more deep dives into groundbreaking AI research.

Ashley: You can follow us on your favorite podcast platform and visit Hugging Face to stay updated on the latest papers.

Until next time, stay curious and keep exploring.

Evan: Thanks for listening to Daily Paper Cast.

Ashley: Goodbye!