SoL-Pi: Recursively Scaling Auto-Research Loops for Efficient Agent Harness

Evan: Welcome to Daily Paper Cast.

Ashley: In today's episode, we're diving into a paper from the Hugging Face daily paper list for September 18, 2026, which has garnered 33 upvotes.

Evan: The paper we're discussing today is titled 'SoL-Pi: Recursively Scaling Auto-Research Loops for Efficient Agent Harness'.

Ashley: It's authored by Haozhe Liu and Tian Ye, with Ruihua Zhang as the corresponding author.

They're all from NVIDIA.

Evan: Let's jump straight into the Introduction section.

Ashley: The paper begins by highlighting the advances in foundational models that enable agents to tackle more open-ended tasks over extended periods with minimal supervision.

Evan: Right.

They mention this shift supports a range of applications, from autonomous research and software engineering agents to self-evolving personal assistants and early forms of recursive self-improvement, or RSI.

Ashley: The main focus is on task-level token efficiency, which becomes a crucial concern as agents operate over these longer horizons.

Evan: Most existing work on efficiency has focused on lowering the cost per token through faster attention kernels, serving infrastructure, model compression techniques, or using cheaper models.

Ashley: Correct.

However, this paper explores an orthogonal direction: improving token use through the agent harness, which mediates the interactions between the model and its environment.

Evan: The authors argue that harness-level optimization can improve efficiency without additional model training, complementing both infrastructure and model-level approaches.

Ashley: But optimizing a harness is challenging because the components like tool use, context management, verification, delegation, recovery, and termination are tightly coupled.

Evan: So a change that's beneficial in one area might cause issues elsewhere or shift token costs to later stages of execution.

Ashley: Exactly.

In practice, developing an optimized harness requires substantial human effort to scrutinize long execution traces, identify recurring failure modes, and translate these observations into code changes.

Evan: To accelerate this process, the authors adopt an RSI-inspired approach in which an AI optimizer iteratively enhances the agent harness for token efficiency.

Ashley: Their SoL-Pi workflow involves a research AI observing execution traces from a separate agent running the base harness, proposing candidate changes, and testing them in prepared research environments.

Evan: These candidates go through capability and efficiency checks to determine which are retained.

Development results then guide subsequent iterations.

Ashley: The paper mentions recent advancements in automated harness improvement, such as Meta-Harness and Recursive Harness Self-Improvement.

Evan: However, an important challenge with these methods is that they can overfit the tasks used during the search phase and offer only marginal gains on unseen tasks.

Ashley: Precisely.

To address this, SoL-Pi organizes its autonomous research as a broad-to-deep funnel that separates candidate development from held-out validation and scales through isolated search lineages.

Evan: The system is guided by three key principles: breadth and depth to expand hypothesis coverage, independent validation to prevent task-specific solutions, and scalable orchestration to support isolated search lineages.

Ashley: By applying these principles, the authors scaled AI-led auto-research across approximately 150 proposed directions and 500 executable environments, resulting in over 3,000 runs and more than 60,000 agent-environment interactions.

Evan: Their extensive search led to the identification of four mechanisms that together form the SoL-Pi system.

Ashley: On the EdgeBench evaluation, SoL-Pi achieved performance comparable to Pi across models like GPT-5.6 Sol and Opus 5 while significantly reducing token traffic and API costs.

Evan: Now that's impressive.

They report reductions in recorded token traffic by 44.7 to 49.0 percent and API costs by about one third.

Ashley: And these improvements translate to estimated hourly savings of between $8.75 and $13.50 compared to native Codex and Claude Code harnesses, and between $4.36 and $5.71 relative to Pi.

Evan: That sums up the Introduction section of the paper—making a compelling case for this novel approach to harness optimization for token efficiency.

Evan: Let's move on to the Method section, Ashley.

Can you walk us through the SoL-Pi system's approach to harness optimization for token efficiency?

Ashley: Of course, Evan.

The SoL-Pi system approaches harness optimization by framing it as an RSI-inspired search for reusable efficiency mechanisms.

Essentially, they start with a broad pool of agent-generated hypotheses.

Evan: Interesting.

How does the research agent analyze these hypotheses?

Ashley: The research agent begins by analyzing execution trajectories from the base harness to identify recurring sources of overhead.

It then builds an idea pool of candidate harness changes.

Evan: So from there, the candidates undergo some kind of testing, right?

How does that process work?

Ashley: Exactly.

The candidates are tested in development environments.

To qualify for acceptance, each mechanism must improve efficiency beyond a single development environment while preserving the agent’s ability to complete the required tasks.

Evan: But how does SoL-Pi ensure these candidates are effective across diverse environments?

Ashley: SoL-Pi employs a broad-to-deep harness search, which allocates research effort in two stages.

The outer stage explores a broad pool of mechanism hypotheses, while the inner stage develops selected hypotheses independently.

Evan: What do they mean by 'independent development' of hypotheses?

Ashley: Well, independent development means that unpromising directions can be terminated without affecting other experiments.

This prevents failures in one area from impacting the entire process.

Evan: Got it.

How do they identify these sources of overhead in the first place?

Ashley: Before they assign rollout budgets, Oracle Analysis examines existing development trajectories to identify avoidable work in the base harness.

Each selected direction must pinpoint a concrete source of overhead and propose a harness change to address it.

Evan: This sounds quite comprehensive.

How do they ensure that only beneficial changes are retained?

Ashley: Each proposed change undergoes several iterations of the conventional auto-research cycle.

This includes proposing a change, implementing it, running a fixed experiment, inspecting the result, and retaining, revising, or discarding the candidate.

Evan: And they combine this cycle with something called the Ralph Loop, right?

Ashley: Correct.

The Ralph Loop adds an iterative implementation cycle.

The implementer refines the candidate to meet explicit completion criteria, followed by independent review before evaluation.

Failed reviews trigger revisions.

Evan: So each experiment operates as a disposable instance of a shared skill template?

Ashley: Yes, precisely.

Each experiment sets its parameters and runs to completion, retaining the candidate and evidence while discarding modified orchestration code.

This approach helps scale the search across broad hypothesis pools and isolated loops.

Evan: I noticed they use separate environments for mechanism discovery and transfer evaluation.

Could you explain more about that?

Ashley: They utilize 535 executable environments for the search set.

This includes 495 repository tasks derived from GitHub issue-pull request pairs and 40 synthetic tasks with executable success verifiers.

Evan: What is the benefit of using repository-derived environments in this process?

Ashley: Repository-derived environments pair a GitHub issue with its pre-fix repository state and offline dependencies.

They use the accepted patch and change history as a reference trajectory, ensuring the test fails before the patch and passes afterward.

Evan: And what about the verifier-driven environments?

Ashley: Verifier-driven environments generate an executable verifier defining success, then build a task environment around it.

This approach allows multiple valid solution paths without needing a reference trajectory.

The synthetic environments primarily use a Terminal-Bench-2 style verifier interface.

Evan: I see.

This combination really broadens the scope of their search.

Now, what mechanisms did they discover from this exhaustive search process?

Ashley: Their search yielded four reusable mechanisms: Action Fusion, Online Context Compact, ObservationPack, and Evidence-Preserving Reducer.

These target different areas like action execution, context management, observation storage, and delegated reading.

Evan: Can you elaborate on how these mechanisms work?

Ashley: Sure.

Action Fusion combines file edits and follow-up commands into one tool request, reducing unnecessary model round trips.

Online Context Compact reevaluates when to compact the context based on plan-step completion, adjusting to save tokens.

Evan: And what about ObservationPack and Evidence-Preserving Reducer?

Ashley: ObservationPack archives large tool outputs locally after the first few requests, replacing them with stable handles and excerpts for efficiency.

Evidence-Preserving Reducer compresses build and test logs, retaining key evidence in a compact form verified through a deterministic process.

Evan: They integrated these mechanisms into Pi and tested them thoroughly, right?

Ashley: Yes, Evan.

Each mechanism was implemented as an extension to Pi and included features like optional follow-up commands for Action Fusion and context caching gates for Online Context Compact.

Evan: Before their held-out evaluation, they freeze the source, configuration, metrics, and acceptance rule to ensure a consistent testing ground.

Ashley: On EdgeBench, SoL-Pi demonstrates significant token traffic reduction and API cost savings while maintaining performance levels comparable to Pi.

This validates the efficacy of their discovered mechanisms.

Evan: This concludes the Method section of the 'SoL-Pi: Recursively Scaling Auto-Research Loops for Efficient Agent Harness' paper.

Now that we understand their approach and methodologies, let's delve into the experimental evaluation details next.

Evan: Let's dive into the Experimental Evaluation section, Ashley.

How did they assess SoL-Pi's performance?

Ashley: They evaluated SoL-Pi across multiple benchmarks, including EdgeBench, Terminal-Bench 4, the International Mathematical Olympiad 2026, and a kernel-optimization benchmark.

Each evaluation aimed to measure average score, token traffic, API cost, and token efficiency.

Evan: Could you break down the results for EdgeBench first?

Ashley: Sure.

On EdgeBench, SoL-Pi was compared with native harnesses like Codex running GPT-5.6 Sol, and Claude Code with Opus 5, as well as Pi.

SoL-Pi achieved an average score of 42.0, which is roughly comparable to Pi's 44.8, while using 49.0% fewer tokens, reducing token traffic from 2.1538 billion to 1.0990 billion tokens.

Evan: That's a significant reduction.

How about the cost savings?

Ashley: In terms of API cost, SoL-Pi brought down the expense from $1,339 to $894, which is a 33.2% reduction.

The complete stack of SoL-Pi not only maintained efficiency but also ensured that the improvements transitioned well between GPT-5.6 Sol and Opus 5.

Evan: So, the results show good generalization to a different model backend as well?

Ashley: Exactly.

When applied to Opus 5, SoL-Pi managed to retain 94.3% of Pi's performance while reducing token traffic by 44.7% and API costs by 33.5%.

This shows SoL-Pi's improvements are not limited to a single model or environment.

Evan: What's the next benchmark they used?

Terminal-Bench 4, right?

Ashley: Yes, on Terminal-Bench 4, which uses 63 CPU-only tasks, both Codex and Pi solved 18 tasks.

SoL-Pi solved 15, but it achieved this while reducing total model cost by 26.3%, dropping from $286.45 with Pi to $211.12 with SoL-Pi.

The cost per solved task also went down from $15.91 to $14.07.

Evan: So it's clear that even though SoL-Pi solved fewer tasks, it did so much more efficiently.

Ashley: Exactly.

Moving on to the IMO 2026 problems, they used GPT-5.6 Sol at high settings to formalize and verify solutions.

Here, SoL-Pi matched Pi in solving three out of six problems but at a total model cost that was the lowest among Codex, Pi, and SoL-Pi.

The cost per passed problem was $20.90 for SoL-Pi, compared to $22.89 for Codex and $25.32 for Pi.

Evan: This indicates that SoL-Pi's harness improvements offer substantial cost savings even in highly formal and challenging tasks like the IMO.

Ashley: Lastly, they also tested SoL-Pi in a multi-agent kernel-optimization scenario, measuring the performance in simulated machine cycles.

Evan: What did they find in that setup?

Ashley: They ran three configurations: a single Codex agent, a Codex coordinator with 20 Pi baseline workers, and a Codex coordinator with 20 SoL-Pi workers.

Across a two-hour run, the SoL-Pi worker setup managed to reduce API cost by 26.8% while achieving 1,127 cycles, compared to 1,333 cycles and $82.12 for Pi, and 1,366 cycles at $39.20 for the single agent configuration.

Evan: So, SoL-Pi helps not just in individual tasks but also in collaborative and multi-agent setups.

Ashley: Right.

The SoL-Pi-enabled swarm and individual agent managed to pass all speed thresholds, showcasing the harness system's capability and effectiveness.

Evan: This thorough evaluation highlights the robustness and versatility of the SoL-Pi system.

Ashley: Yes, from reducing token traffic and API costs on diverse benchmarks to maintaining task performance and extending the benefits to multi-agent systems, SoL-Pi demonstrates its potential across various AI tasks.

Evan: That's the end of the Experimental section.

Evan: Next, let's discuss the Related Work section of the paper.

Ashley, can you take us through it?

Ashley: Certainly, Evan.

The Related Work section provides a contextual foundation, linking SoL-Pi to various strands of research in AI agent harnesses and automated design.

Evan: What are the primary areas of related research they touch upon?

Ashley: The authors delve into agent harnesses, automated agent design, harness optimization, and token-efficient agents, highlighting key developments and methodologies that have influenced their work.

Evan: Let's start with agent harnesses and automated agent design.

What does the paper highlight here?

Ashley: The paper references studies that show an agent's behavior isn't solely dependent on its underlying model but also significantly influenced by the harness that presents state, exposes actions, and processes feedback.

For example, the SWE-agent research demonstrated how altering the agent-computer interface around a fixed model can impact software-engineering performance.

Evan: That's quite intriguing.

What else do they discuss in terms of automated agent design?

Ashley: Automated agent design focuses on various optimization targets such as prompt refinement and code structure.

Methods like GEPA optimize prompts through execution trajectory reflection, while ADAS formulates agent design as a code search problem.

AFlow and AgentSquare explore workflow structures and modular agent components, respectively.

Evan: These approaches sound like they aim to create more efficient or effective agents.

How does this connect to SoL-Pi's broader goals?

Ashley: Exactly.

SoL-Pi builds on this idea by applying recursive self-improvement principles from theoretical concepts like the Gödel Machine to discover reusable efficiency improvements across multiple harness components, while keeping the underlying model unaltered.

Evan: Alright, let's move on to harness optimization.

What does the paper say about recent advances in this area?

Ashley: The paper notes several methods that use execution feedback to optimize harness code and configurations.

Examples include AutoHarness, which synthesizes environment-specific code guards, and Recursive Harness Self-Improvement (RHI), which refines prompt-level specifications for individual task loops.

Evan: And how does the paper address the generalization issue, which is critical for these optimization techniques?

Ashley: A key challenge highlighted is the limited transferability of evolved harnesses to unseen tasks.

Methods evaluated by Wang et al.

exhibited only marginal gains on held-out tasks, thus emphasizing the necessity for separating search feedback and final evaluation to prevent overfitting.

Evan: This seems central to SoL-Pi's design.

How does the SoL-Pi system cope with this issue?

Ashley: SoL-Pi addresses this by organizing autonomous research into a broad-to-deep funnel, refining mechanisms through independent development and keeping validation strictly separate from the search process.

This setup aims to discover harness improvements that generalize well beyond the development environment.

Evan: Next, let's touch on token-efficient agents.

What are the notable methodologies here?

Ashley: Reducing token overhead is critical for long-horizon tasks.

Approaches like AgentDiet remove redundant trajectory content, and ACON optimizes context compression.

Context-Folding and AgentFold manage context through trajectory folding and compression, while Context as a Tool exposes context maintenance as an explicit action.

Evan: How do these methods relate to what SoL-Pi aims to achieve?

Ashley: SoL-Pi extends these methodologies by automatically discovering and selecting harness components that reduce token usage while preserving task performance.

It integrates diverse solutions across action execution, context management, observation handling, and delegated reading into a unified system.

Evan: It sounds like SoL-Pi leverages these research insights to build a more efficient and robust harness system.

Ashley: The paper builds on significant contributions in these fields, demonstrating how harness optimization can lead to substantial efficiency gains and generalization across models and tasks.

Evan: That wraps up our discussion on the Related Work section.

Evan: Alright, Ashley, let's wrap up this episode by summarizing the paper's key contributions and takeaways.

Ashley: At its core, the 'SoL-Pi: Recursively Scaling Auto-Research Loops for Efficient Agent Harness' paper proposes an innovative approach to harness optimization for token efficiency.

By applying recursive self-improvement principles, the system scales auto-research loops across a broad range of environments, ultimately discovering reusable harness improvements that enhance efficiency without additional model training.

Evan: The paper introduces four key mechanisms: Action Fusion, Online Context Compact, ObservationPack, and Evidence-Preserving Reducer.

Each of these mechanisms targets different aspects of the agent’s workflow, from action execution to context management and observation handling, contributing to significant token savings.

Ashley: On extensive benchmarks like EdgeBench, Terminal-Bench 4, and IMO 2026, SoL-Pi demonstrated reduced token traffic and API cost while maintaining or improving task performance.

The system achieved up to 49% reduction in token use and up to 33% in cost savings, evidencing its efficiency across various models like GPT-5.6 Sol and Opus 5.

Evan: Moreover, SoL-Pi's ability to generalize its improvements across different model backends underscores its robustness.

This not only validates the effectiveness of the discovered mechanisms but also shows the system's potential for broader applications in AI automation and multi-agent systems.

Ashley: In essence, SoL-Pi leverages automated research and recursive improvement to offer substantial advancements in token efficiency for agents.

This marks a significant step towards scalable and cost-effective AI systems.

Evan: That's all for today's episode.

We hope you found this discussion insightful and engaging.

Ashley: Thank you for joining us on Daily Paper Cast.

Don't forget to tune in again tomorrow for more deep dives into exciting papers from the world of AI and NLP.

Evan: Until next time, keep exploring and stay curious!