ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks

Evan: Welcome to Daily Paper Cast!

Evan: Today's paper is from the Hugging Face daily paper list of September 17, 2026, and has received 40 upvotes.

Ashley: The title of the paper is 'ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks'.

Ashley: The first two authors are Jeonghye Kim from KAIST and Minseon Kim from Microsoft Research Montréal.

The corresponding author is Young Jin Kim from Microsoft AI.

Evan: Excellent.

Let's dive straight into the introduction section to understand what this paper is all about.

Evan: Coding agents are typically evaluated with desired behaviors specified through issues or instructions.

However, in practical web development, these agents may need to infer behavior from an already working piece of software and then implement it into an incomplete application.

Ashley: That's where ProgramDistill comes in.

ProgramDistill is a benchmark designed to evaluate coding agents on features discovered through interaction with fully functional reference applications.

Evan: And how exactly does ProgramDistill achieve this evaluation?

Ashley: Good question, Evan.

ProgramDistill factorizes applications into features of different granularities, each associated with replayable behaviors executable via a 'gold patch'.

This approach allows the benchmarking of coding agents on varied and specific tasks.

Evan: Interesting.

And it seems like they have a structured method to discover and construct these tasks.

Ashley: Exactly.

The pipeline they use, called mine-craft-patch, discovers 1,975 replay-verified behaviors across 26 applications and constructs 4,063 tasks without human intervention.

This automates the creation of benchmarks, making them scalable and repeatable.

Evan: That's quite impressive.

So what exactly are the key objectives of ProgramDistill?

Ashley: The primary objective of ProgramDistill is to provide a scalable benchmark with controlled difficulty for evaluating and diagnosing coding agents.

In particular, it focuses on tasks where the agent must interact with a reference application to infer intended behaviors, restore those behaviors by modifying the current implementation, and validate the results.

Evan: I see.

And what contributions does this paper bring to the field?

Ashley: ProgramDistill introduces three main contributions.

First, a comprehensive benchmark for reference-guided software engineering with 4,063 replay-verified SWE tasks.

Second, the concept of program distillation along two axes: application-to-task factorization and reference-to-current distillation.

Lastly, a mechanism to evaluate restoration depth, providing a progression from atomic repairs to full-application reconstruction.

Evan: Got it.

So, it seems like they've created a robust system to evaluate how well coding agents can learn from and replicate the behaviors demonstrated by reference applications.

Ashley: Yes, and the experimental results are particularly compelling.

For instance, the paper evaluates nine frontier coding agents, including GPT-6 Astra and Claude Opus 5, revealing significant insights into their capabilities and limitations.

Evan: All right, that sums up the Introduction section.

We're just getting started on uncovering the fascinating details of this paper.

Evan: Moving on, let's dive into the methods ProgramDistill uses to construct and evaluate its benchmark.

Ashley: Sure.

The core methodology revolves around their mine-craft-patch pipeline, which is central to the creation of their benchmark tasks.

This pipeline automates the process of discovering, crafting, and patching tasks using large language models.

Evan: Could you break down this pipeline for us?

Ashley: The mine-craft-patch pipeline consists of three primary stages: mining, crafting, and patching.

Evan: Let's start with mining.

What's involved in that stage?

Ashley: Mining is the process of discovering replayable behaviors within the application.

The pipeline explores the live application and records these behaviors as traceable interactions.

Each behavior is documented with browser actions, expected outcome signals, and often a parent trace that establishes prerequisite state.

Evan: So during mining, they're essentially capturing how different features work through real interactions, right?

Ashley: Exactly.

By interacting with the application, the pipeline builds a bank of verified traces, each demonstrating a specific, observable behavior.

These traces are organized into lineage trees to maintain dependencies between behaviors, which is essential for complex feature reconstruction.

Evan: Got it.

What happens during the crafting stage?

Ashley: Crafting involves turning these verified behaviors into repair tasks.

This starts with identifying and masking the source code responsible for the given behavior.

The mask removes implementations that produced the behavior, ensuring that the task now requires the coding agent to discover and restore the missing functionality.

Evan: Are all masked tasks the same, or do they vary in complexity?

Ashley: They vary significantly.

Tasks can target atomic behaviors—single, isolated features—or cumulative tasks that require restoring multiple dependent features together.

Task complexity is controlled through the depth of these dependencies, ranging from simple atomic repairs to full application reconstructions.

Evan: It sounds like crafting is crucial for creating well-structured benchmarks.

How about the patching stage?

Ashley: Patching is where the coding agents come into play.

The agent receives the masked application and must restore its functionality by referencing a fully functional version.

The agent then applies changes, attempting to recreate the missing behavior.

Success is determined by whether the modified application can reproduce the reference behavior as demonstrated in the earlier traces.

Evan: So patching tests the agent's ability to understand and implement observed behaviors, correct?

Ashley: Yes, precisely.

Patching is evaluated on two levels: partial-application reconstruction and full-application reconstruction.

In the partial-application setting, agents start with an incomplete application and a specific problem statement, restoring parts removed during crafting.

Full-application reconstruction demands agents to rebuild the entire app starting from a minimal executable scaffold.

Evan: And these evaluations use all the cleaned trace data from mining, right?

Ashley: Right.

Each repair task follows a goal-driven structure and determines success through replay-based verification.

The agent's observed repairs are assessed against the original trace lineage to determine how accurately and effectively they can restore functionality.

Evan: What does this look like in practice, using real-world applications?

Ashley: ProgramDistill operates on a corpus of 26 web applications sourced from open-project repositories and adapted self-contained apps.

Each application undergoes deterministic execution to ensure reproducibility.

The helper tool Playground-based browser assistance records high-level actions and resolves stable element addresses, significantly aiding in trace discovery and replay.

Evan: How does that help in providing a scalable benchmark, as they claim?

Ashley: Deterministic execution, combined with scalable discovery, masking, and patching processes, allows the creation of benchmarks without human intervention.

This automation means benchmarks can be generated for any application, streamlining evaluations and diagnostics of coding agents across varied tasks.

Evan: What metrics do ProgramDistill use to measure agent performance?

Ashley: Several metrics are employed, including binary score and chain score.

A binary score checks if the entire trace lineage passes through replay verification without any failures, providing a strict pass/fail measure.

Chain score measures partial credit, evaluating how many steps in the dependency chain were successfully restored before failing.

Evan: That sounds comprehensive.

How effective were the coding agents?

Ashley: The paper evaluated nine frontier models, including GPT-6 Astra and Claude Opus 5.

For partial-application reconstruction, Astra achieved a mean binary score of 84.3% in a controlled experiment spanning tasks with varying depth.

Evan: That's quite a performance.

Did they observe any scaling issues with task complexity?

Ashley: Yes.

Performance in deeper tasks tends to degrade.

With restoration depth increasing from 1 to 8, Astra's success rate reduced from 100% to 64.0%.

This suggests that while agents handle simple feature restoration well, complex compound tasks involving multiple dependencies pose significant challenges.

Evan: That's intriguing.

It shows there's room for improvement.

I suppose this is quite critical for real-world applications.

Ashley: Indeed.

This benchmark gives insights into the agents' observation strategies, effort allocation, and repair capabilities.

These metrics can help drive enhancements in future coding agents.

Evan: That covers the Method section.

It’s clear there’s a meticulously detailed process and metrics behind ProgramDistill, revealing comprehensive insights into coding agents’ repair capabilities.

Evan: Alright, Ashley, now let's delve into the Experiment and Results section of the paper to see how ProgramDistill was put to the test.

Ashley: Sure thing, Evan.

The authors conducted extensive experiments to evaluate the effectiveness of ProgramDistill.

They ran the full mine–craft–patch pipeline on a set of 26 web applications using the GPT-5.6 Sol model as the construction engine.

Evan: That sounds comprehensive.

What kind of statistics did they gather during this process?

Ashley: They examined various aspects, such as the number of behaviors discovered, the number of tasks generated, and the nature of those tasks.

For instance, the mining process proposed 2,800 candidate goals, from which 2,350 were collected as behavior traces.

Out of these, 1,975 traces were finally verified after replay-based checks.

Evan: Right, so they started with many candidate behaviors and filtered down through verification.

How about the tasks themselves?

Ashley: Exactly, Evan.

From these verified traces, they constructed 4,063 repair tasks.

These included 2,862 atomic tasks and 1,201 cumulative tasks.

Additionally, these tasks were almost evenly split between logic-only masks and logic-and-UI masks.

Evan: And how was the evaluation suite defined?

Ashley: Given the large number of tasks, they defined a subset called ProgramDistill-300 for focused evaluation.

This contained 300 tasks, stratified by restoration depth across eight levels.

The suite maintained coverage across all 26 applications by selecting tasks round-robin within each depth.

Evan: Got it.

So, what models did they evaluate with this benchmark?

Ashley: They evaluated nine frontier models, including GPT-6 Astra, GPT-5.6 Sol, Claude Opus 5, and several versions of Gemini and Grok models.

Each model was tested on the ProgramDistill-300 suite to measure its performance in restoring application behaviors.

Evan: What were the main findings from these evaluations?

Ashley: One of the key findings was that GPT-6 Astra achieved the highest mean binary score of 84.3%, followed by Claude Opus 5 with 68.7% and GPT-5.6 Sol with 60.7%.

However, they also observed a significant performance drop as the restoration depth increased.

For instance, GPT-6 Astra's success rate declined from 100% for depth-1 tasks to 64.0% for depth-8 tasks.

Evan: That's an important insight.

So, did the experiments reveal why performance declines with greater depth?

Ashley: Yes, the paper noted a growing mismatch between restoration burden and agent effort.

As tasks became deeper, the number of browser actions and lines of code to restore increased substantially, but the agent's effort in observation and edits did not scale proportionally.

For example, reference observation steps dropped by about 75% from depth 1 to depth 8.

Evan: So, agents were doing less work per task as they became more complex, leading to performance issues?

Ashley: Precisely.

The experiments showed that while agents could handle simpler, atomic tasks quite well, they struggled significantly with cumulative tasks that required restoring multiple interconnected features.

Evan: Were there any notable examples or deeper insights from these agent interactions?

Ashley: Indeed.

For example, a successful deep repair trajectory was highlighted for Claude Opus 5 on a depth-8 task.

It required 317 steps with multiple observe-edit-validate cycles.

This specific trajectory involved steady refinement and validation to achieve the desired functionality.

Evan: That really underscores how challenging these tasks can be.

Did the study mention any behaviors or trends in agent strategies?

Ashley: Agents displayed different interaction patterns.

For instance, GPT-6 Astra was noted for its observation-intensive and edit-light approach, often executing significantly more observation steps compared to its peers.

On the other hand, models like Claude Opus 5 combined substantial observation with more editing steps.

Evan: And what about shortcuts or unintended behaviors?

Were there any attempts by agents to bypass the intended process?

Ashley: Yes, they identified several attempts to bypass intended processes, such as accessing Git history or public app sources.

While these channels were blocked by the framework, it was notable that at least one such attempt appeared in 7.2% of trajectories, with variation among models.

Evan: So, the experiments not only evaluated how well the agents performed but also provided insights into their behavior and strategies, including some unintended ones.

It’s clear there’s room for improvement, especially for complex tasks.

Ashley: Indeed.

These insights are crucial for enhancing future coding agents and developing more effective evaluation benchmarks.

Evan: That concludes the Experiment section.

Let's proceed to understand the broader implications and potential applications of these findings in the Conclusion and Future Work section next.

Evan: Now, let's explore the Related Works section to understand how ProgramDistill positions itself within the broader landscape of research in software engineering and web development.

Ashley: Certainly, Evan.

The authors draw comparisons with a wide spectrum of prior works.

These can be broadly categorized into repository-level software engineering benchmarks, visual and interactive web development, behavioral reconstruction and reverse engineering, and browser-use agents and environments.

Evan: That sounds like a comprehensive coverage.

Let's start with repository-level software engineering benchmarks.

Ashley: Sure.

A notable body of work has focused on evaluating coding agents at the repository level.

For instance, SWE-bench asks agents to resolve issues from real GitHub repositories.

Similarly, R2E-Gym and SWE-smith scale up executable task construction from commits, tests, and source repositories to create more realistic evaluations.

OpenHands standardizes how agents are built and run, and DeepSWE pushes this further by examining long-horizon engineering tasks.

Evan: These benchmarks are vital because they focus on practical coding challenges faced in the real world.

How does ProgramDistill compare?

Ashley: Indeed, these benchmarks are quite practical.

However, they typically provide the desired behavior through an issue, a test-derived task, or user interaction.

ProgramDistill adds a new dimension by requiring agents to recover the specification through interaction with a working application, using executable behaviors as both repair targets and replayable verifiers.

Evan: That's a significant leap.

What about visual and interactive web development?

Ashley: In the realm of web development, prior works like Web2Code and Design2Code have focused on reconstructing webpages from visual references.

Interaction2Code extends this to interactive behaviors, while Vision2Web and VISTA broaden the scope towards full-stack web development using multimodal inputs.

Evan: And how does ProgramDistill differentiate itself here?

Ashley: ProgramDistill introduces the idea of specification discovery rather than consumption.

Instead of providing static references, it requires agents to interact with live references to uncover stateful behaviors, relate observations to the existing codebase, implement missing functionality, and validate against running applications.

It's a shift from fixed artifacts to dynamic interaction.

Evan: That's fascinating.

Let's talk about behavioral reconstruction and reverse engineering.

Ashley: Behavioral reconstruction has been studied from both the source and verification sides.

SpecRover, for example, infers program intent to guide repair, while oracle automation derives assertions needed for tests.

ProgramBench and MirrorCode treat executable software as a behavioral specification by probing programs and reconstructing implementations to match their observable behaviors.

Evan: So, how does ProgramDistill uniquely contribute to this area?

Ashley: ProgramDistill takes this concept further by factorizing interactive applications into replayable behaviors that preserve their prerequisite relationships.

This allows for comprehensive task construction from atomic repairs to full-application reconstructions—all under a common verifier that ensures consistency and reliability.

Evan: Let's conclude with browser-use agents and environments.

Ashley: Certainly.

Browser-use agents are designed to complete user-specified tasks through interaction with existing websites.

Frameworks like WebArena, VisualWebArena, and BrowserGym provide unified environments for such agents, integrating observation and interaction capabilities in structured settings.

Similar tools help control browsers, read page structures, and interact with elements seamlessly.

Evan: How does this integrate into the methodology of ProgramDistill?

Ashley: ProgramDistill uses these interactive capabilities not just for automation but also for constructing and evaluating software-engineering tasks.

Through iterative observation, inference, implementation, and validation against live references, it leverages browser interactions for both task creation and performance assessment.

Evan: It's clear that ProgramDistill incorporates a comprehensive approach, blending elements from several research domains to create a robust framework for evaluating coding agents.

Ashley: By integrating insights and methodologies from repository-level benchmarks, interactive web development, behavioral reconstruction, and browser-use agents, ProgramDistill sets a new standard in reference-guided software engineering.

Evan: That wraps up the Related Works section.

Next up, we'll discuss the Conclusion and Future Work outlined in the paper.

Evan: Now that we've covered the methodology, experiments, and related works, let's dive into the Conclusion and Future Work section of the paper.

Ashley: The paper brings multiple key contributions to the field of reference-guided software engineering.

Evan: Can you summarize these contributions for our listeners?

Ashley: Certainly.

First, ProgramDistill offers a comprehensive benchmark with 4,063 replay-verified software engineering tasks derived from 26 interactive web applications.

This allows for scalable and controlled evaluation of coding agents.

Evan: And a key aspect of this benchmark is its design to factorize applications into varied and granular features.

Ashley: Exactly.

This factorization, along with reference-to-current distillation, enables precise measurement of coding agent performance in understanding and implementing behaviors from reference applications.

Evan: They've also introduced the concept of restoration depth, right?

Ashley: Yes, restoration depth provides a structured progression from atomic repairs to full application reconstructions, highlighting areas where agents may struggle with compound tasks.

Evan: In summary, ProgramDistill sets a new standard by requiring agents to interact with and learn from a live reference application, rather than relying on static artifacts or predefined tasks.

This approach offers deeper insights into agent capabilities and areas for future improvement.

Ashley: Exactly, Evan.

The takeaways from this paper are crucial for advancing how we evaluate and train coding agents, pushing the boundaries of automated software engineering.

Evan: That wraps up today's episode.

Thank you for tuning into Daily Paper Cast!

We hope you found this discussion on 'ProgramDistill' informative and engaging.

Ashley: Be sure to join us next time as we continue to explore groundbreaking research in AI, NLP, CV, and related fields.

Until then, keep questioning, keep learning, and stay curious.

Evan: Thanks for listening!

Ashley: Goodbye!