An Empirical Study of Harness Design for Coding Agents

Evan: Welcome to the Daily Paper Cast.

Evan: Today's paper comes from the Hugging Face daily paper list of September 18, 2026, and it has garnered 30 upvotes.

Ashley: The paper we're discussing is titled: 'An Empirical Study of Harness Design for Coding Agents'.

Ashley: It is authored by Run-Ze Fan and Zihao Zhang, with the corresponding author being Run-Ze Fan from the University of Massachusetts Amherst.

Evan: Large language models, or LLMs, are increasingly taking on real software-engineering tasks autonomously.

We're talking about things like closing GitHub issues or completing end-to-end terminal tasks.

Ashley: Exactly.

These breakthroughs are made possible by coding harnesses.

Essentially, a coding harness is a software layer with various components that guide how these models behave.

You have aspects like planning the tasks, deciding the operations that become executable actions, and managing interaction history within a finite context window.

Evan: So, it's not just about the core model; the harness itself plays a significant role in performance?

Ashley: Precisely.

Previous studies have often evaluated these harnesses as complete systems, but this confounds the effectiveness of the individual components.

A key research question then arises: Are these individual components generally useful across different settings, or is their effectiveness context-specific?

Evan: Hmm, that's interesting.

What are the main objectives of this study?

Ashley: The study aims to dissect the impact of three central components of coding harnesses: planning, action space, and context management.

By holding the model and the overall execution loop constant, the researchers vary these three components to evaluate their independent effects.

Evan: Got it.

And why focus on these specific components?

Ashley: Well, prior systems have identified these as the key elements necessary for the long-term performance of coding agents.

Planning maintains task progress, the action space determines how model intents are translated into executable operations, and context management decides which interaction history stays available as the tasks grow.

Evan: Okay, let's get into the contributions then.

What are the unique angles this paper brings?

Ashley: They propose a lightweight coding harness with a fixed execution loop while varying the planning, action space, and context management components.

For context management in particular, they define and evaluate five distinct strategies involving elision, recall, and summarization.

Evan: And they tested this on multiple models and benchmarks?

Ashley: Yes, that's correct.

They tested three sizes of the Nemotron-3 model family and Mistral-Medium-3.5 across two long-horizon coding benchmarks: SWE-Bench Verified, which deals with repository-level issue resolution, and Terminal-Bench 2.1, which focuses on end-to-end terminal task completion.

Evan: That should give a fair amount of data to analyze.

So, what were the main findings from this study?

Ashley: There are several key takeaways.

First, context management is most valuable under tight context-window budgets.

Its primary utility lies in preventing the agent from prematurely terminating due to context overflow.

As the context window expands, the benefits become marginal.

Evan: Interesting.

And what about the specifics within context management?

Ashley: The strategy that combined rule-based elision before LLM summarization proved to be the most efficient.

It maintained success rates comparable with other strategies while being more cost-effective.

Interestingly, making the elided content recoverable added machinery that was rarely used and didn't improve accuracy.

Evan: So, a combination approach worked best there.

What changes did they see with planning?

Ashley: Planning also showed some variability.

For weaker models, it served as an accuracy scaffold by extending the execution trajectory long enough to attempt a meaningful edit, though this came at a higher computation cost.

For stronger models, planning primarily helped in reducing cost by avoiding redundant post-edit verification, with little change in accuracy.

Evan: And how did the models fare with predefined tools versus a bash-only interface?

Ashley: Predefined tools were more effective for models with weaker bash proficiency, offering a clear performance boost.

However, bash-capable models were able to operate effectively with a bash-only interface, leading to significant cost reductions, especially on command-line-centric tasks.

Evan: Fascinating.

It seems like the harness design really needs to be tailored based on the model's strengths and the tasks at hand.

Ashley: Precisely.

And that wraps up the Introduction section of this paper.

Evan: Ashley, the introduction gave us a solid overview of the study's objectives and high-level findings.

Now, let's dive into the methods they used.

How did the researchers go about dissecting these harness components?

Ashley: Sure, Evan.

The researchers built a lightweight coding harness from scratch to facilitate controlled experimentation.

Their primary goal was to analyze the individual impact of planning, action space, and context management on coding agents.

Evan: Okay, and what does this lightweight coding harness look like?

Ashley: The harness operates on a fixed execution loop, following a ReAct cycle.

Essentially, this means each turn includes a reasoning step where the model decides what to do, an action step where it performs a task, and an observation step where it reviews the outcome.

Throughout their tests, they kept this loop constant while varying the planning, action space, and context management components.

Evan: I see.

Let's break down each of these components.

Starting with the planning, what did they incorporate here?

Ashley: For planning, the researchers implemented a system where the model maintains a persistent task plan.

If planning is enabled, the model is instructed to define an initial plan before taking any action and to update this plan throughout its tasks.

They wanted to see how this scaffold would help or hinder models with different capabilities.

Evan: Interesting.

And for models where planning is disabled?

Ashley: In those setups, all planning-related prompts, instructions, and tools were removed.

This allowed the research team to isolate the actual effect of having a planning scaffold rather than conflating it with other variables.

Evan: Got it.

What about the action space?

Ashley: The action space defines how the coding agent interacts with its environment.

They had two primary conditions here: the predefined-tool setup and the bash-only setup.

The predefined tools covered a wide array of operations like reading, writing, and editing files, listing and searching directories, web fetching, and running shell commands with specific protocols.

Evan: And what about the bash-only setup?

Ashley: In the bash-only setup, the model had to interact with the environment exclusively through shell commands.

This setup lets bash-capable models leverage their command-line skills, theoretically reducing complexity but potentially increasing efficiency.

Evan: That makes sense.

Now, context management is a bit more involved, right?

Ashley: Exactly.

They explored five context-management tiers.

Here's the breakdown:

Ashley: Tier 0 applied no additional context management and terminated the agent's run if it exceeded the context window.

Tier 1 used elision to remove stale content without storing it externally.

Tier 2 added a recall mechanism, making the elided content retrievable later.

Tier 3 used LLM-based summarization to compress older history without elision.

Finally, Tier 4 combined all these strategies: elision first, then storing elided content externally, and summarization as a last resort.

Evan: That sounds comprehensive.

What context-window budgets did they test against?

Ashley: They evaluated these strategies under four different context-window budgets: 32k, 64k, 96k, and 128k tokens.

This wide range allowed them to understand how each strategy performs under varying constraints.

Evan: And you mentioned there were some specific tools and techniques in place for other aspects?

Ashley: Yes, certainly.

Beyond the primary components, they also implemented several supporting mechanisms like stuck detection, permission handling, and post-edit diagnostics.

All these were held constant across experiments to ensure the focus stayed on the main variables — planning, action space, and context management.

Evan: Alright.

So, they used this harness with multiple models, right?

Ashley: Exactly.

They used three sizes of the Nemotron-3 family — 30 billion, 120 billion, and 550 billion parameters — and also included Mistral-Medium-3.5 with 128 billion parameters for cross-family comparisons.

All models were evaluated on two benchmarks: SWE-Bench Verified and Terminal-Bench 2.1.

Evan: That covers a lot of ground.

How did they run these experiments?

Was there anything special about the setup?

Ashley: For the actual experiments, they leveraged a platform called LangGraph to build the harness, with tasks executed through Harbor, a framework for managing long-running jobs.

Most tasks were capped at 300 steps, with various system-specific constraints like soft and hard context thresholds, verbatim recent window, and maximum tool-result lengths.

Evan: And they measured success rates and costs, correct?

Ashley: That's right.

For each experimental setting, they measured two main metrics: task success rate, which is the fraction of tasks the agent successfully completes, and the mean cost per task, accounting for computational resources used.

Evan: How about their data analysis?

Ashley: Their analysis went beyond simple success rates.

They conducted trajectory-level analysis, which allowed them to see how each component affected task progression, termination behavior, context use, and tool invocation over time.

This granular approach helped explain not just whether a component worked but how and why it worked.

Evan: That's incredibly detailed.

Is there anything else in the methods section worth mentioning?

Ashley: Well, one interesting note is their sophisticated stuck detection mechanism.

It watches for repeating, identical actions and injects reminders to shift the model's approach if it notices these patterns.

This helps minimize wasted cycles and emphasizes adaptive problem-solving over brute force.

Evan: Fascinating.

So essentially, the researchers built a meticulous and highly controlled environment to dissect how these core components affect coding agent performance.

Ashley: Exactly.

And that’s the end of the Methods section.

They've laid a robust groundwork for us to understand how the different components in a coding harness can influence overall outcomes.

Evan: Ashley, the methods used by the researchers were certainly comprehensive.

How did they actually proceed with the experiments, and what were the key results?

Ashley: Alright, Evan.

The researchers conducted their experiments using three sizes of the Nemotron-3 model family and the Mistral-Medium-3.5 model.

They evaluated these models across two main benchmarks: SWE-Bench Verified and Terminal-Bench 2.1, each with distinct challenges.

Evan: Let’s dive into the specifics of the experiments.

What were the primary variables they tested?

Ashley: They focused on varying the context management strategies, planning, and action space.

Specifically, they tested five tiers of context management: no additional compaction, elision alone, elision with recall, summarization alone, and a combined approach.

They also compared predefined-tool and bash-only setups for the action space.

Evan: That's a lot of permutations.

What were their findings regarding context management?

Ashley: Context management proved to be crucial, particularly under tight context-window budgets.

When the window was smaller, context management prevented premature termination by avoiding overflow, thereby improving task success rates.

As the context window expanded, the marginal benefits of context management decreased.

Evan: Which strategy turned out to be the most effective?

Ashley: The combination of elision before LLM-based summarization, referred to as Tier 4, was the most efficient.

It maintained high success rates while being more cost-effective compared to other strategies.

Recoverable elision was rarely utilized and didn’t show significant improvements in task accuracy.

Evan: Interesting.

What did the experiments reveal about the planning component?

Ashley: For planning, its utility varied based on the model’s capabilities.

For the weaker Nemotron-3 30B model, planning acted as an accuracy scaffold, prolonging execution enough to attempt meaningful edits, albeit at higher cost.

The stronger Nemotron-3 550B and Mistral-Medium-3.5 models benefited from reduced redundancy in post-edit verification, which lowered costs without significantly changing accuracy.

Evan: So planning has different impacts depending on model strength.

What about the predefined tools versus bash-only interfaces?

Ashley: For models with weaker bash proficiency, predefined tools significantly boosted performance by offering more structured interactions.

Bash-capable models, on the other hand, optimized well in bash-only setups, resulting in lower costs, especially for command-line-centric tasks.

Evan: Can you elaborate on the trajectory-level analysis they performed?

Ashley: Certainly.

The researchers analyzed how each intervention impacted task progression, termination behavior, context utilization, and tool invocation.

They discovered that context management extended execution trajectories without substantially altering agent behavior, especially under tighter context budgets.

Evan: How did the planning scaffold affect task trajectories?

Ashley: Planning sustained the weaker Nemotron-3 30B model through initial edits, significantly extending its task length from around 5 turns to 40 turns.

For stronger models, planning shortened the trajectory length by more efficiently handling post-edit verification processes, improving cost-effectiveness.

Evan: And the action space?

Ashley: Regarding action space, bash-only setups enabled larger and fewer interactions for bash-capable models.

For instance, Nemotron-3 550B saw fewer repetitive patches and more efficient larger edits, while the weaker models struggled without predefined tools and often terminated early without meaningful edits.

Evan: That's a lot of nuanced details.

How does recall play into all this?

Ashley: Recall was rarely used and did not improve accuracy over elision alone, suggesting that while theoretically useful, in practice, lossless storage mechanisms added unnecessary complexity that models seldom utilized.

Its benefits were most apparent under heavy context pressure.

Evan: Thanks for breaking that down.

So, it seems the optimal harness configuration depends heavily on the model and task specifics?

Ashley: Exactly, Evan.

The choice of harness components should be tailored to the model’s inherent strengths and the nature of the task.

The experiment section highlights that harness design is a conditional problem, requiring nuanced configurations based on various parameters.

Evan: That wraps up the Experiment section perfectly, Ashley.

Let's get ready to discuss the conclusion and broader insights next.

Evan: Ashley, we've delved into the methods and experiments comprehensively.

But grounding this in the broader context of existing research is crucial.

What does the Related Work section highlight?

Ashley: Great point, Evan.

This paper meticulously situates its contributions within the landscape of previous research on coding agents, harness designs, and context management strategies.

Evan: Let's start with coding agents.

How do these systems historically perform, and what specific advancements have recent models brought to the table?

Ashley: The researchers reference several state-of-the-art models designed for coding tasks.

Examples include Qwen3-Coder, Kimi K2.5 and K2.6, GLM-5, and Mistral Medium 3.5.

These models, as noted, have been tailored for tasks like repository-level issue resolution and end-to-end command-line task completion, measured on benchmarks such as SWE-Bench and Terminal-Bench.

Evan: So, coding agents are not new, but their capabilities have been steadily evolving, right?

Ashley: Exactly.

Early on, the focus was on simply generating code.

Now, the emphasis is on creating agents that can integrate into longer, more complex workflows, as highlighted by their performance on real-world benchmarks.

Evan: And how does this paper's focus on harness components fit into that evolution?

Ashley: Well, it’s interesting.

The paper argues that while significant strides have been made, a lot of this progress conflates the capabilities of the model itself with the supporting harness.

A prime example given is that different models perform better with different harnesses, which suggests the supporting infrastructure plays a critical role in overall performance.

Evan: I see.

What does this mean for the design of harnesses specifically?

Ashley: Coding harnesses turn these language models into actionable agents by integrating elements like the control loop, tool interfaces, and context management.

Noteworthy harnesses mentioned are Claude Code, OpenAI's Codex, OpenCode, and OpenHands, with each showing that the harness design substantially impacts performance.

Evan: So, harness design is as crucial as the model's capabilities.

What nuances in harness design are explored in this paper?

Ashley: A number of studies have assessed various harness components, but this paper is distinctive in its thorough, targeted examination of specific harness mechanisms across different models.

By ensuring the harness remains modular, the researchers could isolate the effects of implementation-level planning, workspace action interfaces, and context-management policies.

Evan: Does this mean that the paper’s main focus lies on the individual components of harnesses?

Ashley: Precisely.

Unlike previous studies that often treat the harness as a monolithic unit, this paper explores how different harness components independently impact performance.

This sharply contrasts previous work, indicating that the model is only part of the equation.

Evan: Interesting.

How do they address context management in this broader landscape?

Ashley: Context management has been a pressing issue as tasks grow more complex and lengthy.

Past studies typically rely on static methods like elision or summarization or dynamic methods like reinforcement learning for context management.

This paper uniquely examines these strategies in conjunction and tests them under varying conditions.

Evan: So, they bring a fresh perspective by integrating these strategies?

Ashley: Correct.

By combining approaches — like rule-based elision, external storage, and summarization — and testing them across diverse models and task families, the researchers provide nuanced insights into how context management strategies perform under different scenarios.

Evan: Any major takeaways about dynamic versus static context management?

Ashley: The paper clarifies that while dynamic methods offer potential improvements, they are not universally superior.

The effectiveness largely depends on the task and the model's capacity to handle compressed context information.

Static strategies, especially when layered, can match or surpass dynamic methods when tailored correctly.

Evan: That's quite insightful.

Is there any related work on practical applications or real-world implementations of these harnesses?

Ashley: Yes, there's a growing body of work on integrating these systems into real-world coding environments.

For example, Claude Code and OpenAI's Codex have both seen practical implementations, demonstrating significant utility in real-world software engineering tasks.

Evan: So, real-world application is already underway?

Ashley: Indeed, but this paper aims to push the envelope by demonstrating that understanding and optimizing the harness components can lead to more efficient, powerful coding agents, offering a roadmap for future real-world applications.

Evan: Got it.

So, we can expect future coding agents to be even more nuanced and capable thanks to these continued advancements in harness design.

Ashley: Exactly, Evan.

This research shows that refining harness design is as critical as developing the models themselves, and future systems will likely benefit from this dual focus.

Evan: Thanks, Ashley.

That concludes the Related Work section.

Evan: Ashley, we've covered a lot of ground digging into this paper.

Let's summarize its key contributions and takeaways for our listeners.

Ashley: Certainly, Evan.

At its core, the paper provides a detailed empirical analysis of coding harness components, focusing on planning, action space, and context management.

They built a modular harness from scratch, allowing them to isolate and evaluate the impact of each component across multiple models and benchmarks.

Evan: One major takeaway was regarding context management, right?

Ashley: Yes.

The researchers found that context management is crucial under tight context-window budgets, mainly due to its role in preventing premature task termination.

The strategy that combined rule-based elision with LLM summarization, termed Tier 4, offered the best balance of efficiency and task success rates.

Evan: Planning also showed varied effectiveness depending on the model's strength?

Ashley: Exactly.

For weaker models like Nemotron-3 30B, planning served as an essential scaffold, while for stronger models, it reduced computational costs by streamlining post-edit verification.

This demonstrates the importance of tailoring harness components to the specific capabilities of the model.

Evan: And when it comes to the action space, predefined tools helped weaker models, but bash-only setups were better for stronger ones?

Ashley: That's right.

Bash-capable models performed more efficiently with a bash-only interface due to their proficiency in handling command-line tasks.

Conversely, predefined tools provided a structured environment that benefited models with less bash proficiency.

Evan: So, the overarching message is that harness design must be adaptable based on the model and task specifics.

A one-size-fits-all approach doesn't work here.

Ashley: Precisely.

Each component of the harness plays a significant role, and optimizing them according to the model's strengths and task requirements is essential for maximizing performance.

Evan: Thank you, Ashley, for breaking down this complex paper.

And thank you to our listeners for joining us on this deep dive into coding harness design.

Ashley: We hope you found today’s discussion insightful.

Be sure to join us for future episodes where we continue to explore groundbreaking research in AI and machine learning.

Evan: Until next time, stay curious and keep learning.

Goodbye!