ScienceIDE: Turning World's Scientific Codebase into Agent Learnable Environments

Evan: Welcome to Daily Paper Cast.

Ashley: Today we cover a paper from the Hugging Face daily paper list of September 17, 2026, with 63 upvotes.

Evan: The title of the paper is 'Turning World's Scientific Codebase into Agent Learnable Environments.'

Ashley: The first two authors are Hejia Geng and Zesen Huang, with Yingcheng Wu as the corresponding author.

This research is affiliated with the AItonomy Foundation.

Evan: Alright, Ashley, let's dive into the Introduction section.

Can you give us a brief background to start?

Ashley: Of course.

The paper begins by highlighting the increasing capabilities of language models as they gain more experience from various domains.

For instance, using text data to support chat intelligence and coding tasks through repositories, compilers, and tests.

Evan: So, what differentiates scientific intelligence from these other tasks?

Ashley: While coding tasks specify problems with clear goals and acceptance tests, scientific discovery-oriented tasks are more complex.

They require agents to identify valuable questions, test hypotheses with interventions, and draw lessons from the collected evidence.

Evan: Got it.

How does this complexity translate into practical challenges for learning models?

Ashley: This complexity creates what the authors call a 'scientific experience bottleneck.’ Scientific codes come with heterogeneous toolchains and configurations that make reproducibility difficult.

Scientific validation often depends on physical quantities, numerical tolerances, and unwritten conventions.

Evan: So, how does ScienceIDE propose to address this bottleneck?

Ashley: ScienceIDE aims to overcome this by converting scientific codes into programmable environments for agents.

These environments are designed to encapsulate validated numerical models and executable checks, thus making the experience reusable for agent learning.

Evan: Interesting.

And what exactly are the main contributions of ScienceIDE according to the paper?

Ashley: Great question.

The key contributions of ScienceIDE are threefold.

Firstly, it encapsulates expert-defined scientific cases and acceptance criteria into executable environments.

Secondly, it provides a structured foundation for supervised fine-tuning, reinforcement learning, and evaluation of scientific tasks.

Lastly, it lays the groundwork for integrating agent learning processes with scientific practice.

Evan: That sounds like a comprehensive approach.

How do the authors validate their model?

Ashley: They train models named PhAI-IDE-72B, PhAI-IDE-9B, and PhAI-IDE-4B using the verified scientific interaction trajectories.

The models demonstrate gains in scientific-code repair and perform well on selected general-purpose benchmarks, suggesting that scientific experience contributes positively to broader AI capabilities.

Evan: That's intriguing.

So, to wrap up this section, what are they ultimately hoping to achieve with ScienceIDE?

Ashley: The ultimate aim is to make scientific software a shared substrate for developing scientific intelligence.

This involves creating an integrated workspace where both agent learning and scientific practice can coexist and benefit from one another.

Evan: Thanks, Ashley.

So that covers the Introduction section of this paper.

Next, we'll dive deeper into the methodology behind ScienceIDE.

Evan: Now that we've covered the Introduction, let's discuss the Method section.

Ashley, could you start us off by explaining how ScienceIDE turns scientific expertise into reusable agent experience?

Ashley: Sure, Evan.

According to the paper, ScienceIDE begins by defining scientific responsibilities and acceptance criteria through expert calibration.

Each scientific module within a versioned codebase is transformed into an executable environment.

This involves packaging a module with its runtime, tests, and a private verifier.

Evan: Interesting.

Can you explain what these modules include and how they are verified?

Ashley: Certainly.

Each module holds a coherent scientific responsibility and executable coverage, meaning it handles specific scientific tasks with dedicated input and output specifications.

An agent inspects a pinned upstream revision of the code, builds the source, and runs official tests and examples.

This process reveals output formats, numerical variabilities, and execution hazards.

Evan: So it's more than just compiling and running tests; it's about understanding the nuances of each scientific problem.

How do these modules become environments?

Ashley: Exactly.

After initial inspection, the agent proposes modules based on scientific responsibility.

These are then reviewed by domain experts to ensure they cover all critical paths.

Once approved, the modules are packaged into environments with editable workspaces, checks, and a private verifier.

Evan: What kind of checks are we talking about?

Ashley: The checks include running official tests and ensuring that scientific outputs align with known standards.

They have a pointwise policy for comparing graded values and an invariants policy when a pointwise comparison isn't sufficient.

These checks make sure that the agents adhere to scientific conventions and retain scientific integrity during their tasks.

Evan: How do these environments help in generating tasks for agents?

Ashley: Great question.

Once an environment is set up, task factories generate specific tasks by leveraging reusable authoring procedures.

These shared procedures involve reversible edits, execution, and artifact assembly.

Local rules guide the specific paths and transformations that agents need to follow to complete tasks.

Evan: Interesting.

Can you give us examples of the types of tasks generated?

Ashley: The tasks are divided into seven categories: Acceleration, Repair, Discovery, Reproduction, Integration, Calibration, and Implementation.

For instance, a Repair task might involve fixing a defect identified in one part of the code, while an Acceleration task would focus on optimizing a specific function to run faster.

Evan: How are these tasks validated?

Ashley: Validation ensures that generated tasks are observable, solvable, and provide trustworthy outcomes.

Each task undergoes execution checks and validation within its environment.

Successful completion and adherence to scientific standards are necessary for a task to be accepted.

Evan: So once tasks are validated, how do agents interact with these environments?

Ashley: Agents interact with the environments through a common episode interface.

They receive editable workspaces, inspect code and scientific inputs, make changes, run experiments, and submit their outputs to private verifiers.

This process records actions, observations, rewards, execution status, and resource use to guide learning.

Evan: And how do these interactions contribute to model learning?

Ashley: The interaction data serves multiple purposes.

For evaluation, held-out tasks and fixed budgets measure the agents' success rates.

For supervised fine-tuning, selected trajectories help in training new models.

And for reinforcement learning, it provides online rewards to improve policies.

Evan: Can you tell us more about the environments used for training and validation tasks?

Ashley: The paper reports 64 environments derived from 27 scientific codebases, supporting 2,812 tasks.

Among these, repair and implementation are the most common, contributing to nearly all tasks, with a few acceleration tasks added.

Each environment specifies the workload an acceleration task must speed up, providing a detailed and focused training ground.

Evan: You mentioned earlier about the validation and budgets.

Could you expand on the budget aspects?

Ashley: Budget response profiles are tailored to model families.

They analyze the efficiency of resource usage, indicating how long models take to solve tasks and how success rates align with resource expenditure.

This helps in understanding how resource-efficient each model is while maintaining scientific accuracy.

Evan: Anything notable about how different models performed on these tasks?

Ashley: Yes, the paper highlights that different models show varied success rates and resource efficiencies.

For example, the Fable 5.1 model showed a higher success rate in less time compared to other models.

This detailed analysis helps in refining the training and deployment strategies for these models.

Evan: Thanks for that detailed breakdown, Ashley.

Anything else to add before we wrap up the Methods section?

Ashley: Just to emphasize that the core of ScienceIDE's method is creating a robust framework for agent learning in scientific fields.

By formalizing and validating interactions in these programmable environments, the method ensures that the agents’ learning is both scientifically accurate and practically relevant.

Evan: That wraps up our deep dive into the Method section.

Thanks, Ashley.

Evan: Alright, Ashley.

We've covered the Introduction and Methods sections.

Now, let's delve into the Experiment and Results section of the paper.

How do the authors evaluate the effectiveness of ScienceIDE?

Ashley: The authors employ a comprehensive experimental setup to evaluate both scientific task execution and the benefits of learning from scientific interactions.

They use ScienceIDE-Hard, a validated subset of 85 hard tasks derived from five repository families: PLUTO, Athena++, MITgcm, LAPS, and PHANTOM.

Evan: What types of tasks are included in ScienceIDE-Hard?

Ashley: ScienceIDE-Hard includes 52 repair and 33 implementation tasks.

These tasks require agents to correct defective code, reconstruct missing functionality, and deliver specific scientific outputs that are precisely aligned with private scientific references.

Evan: How do the authors measure success and document the model comparisons?

Ashley: The primary metric for evaluation is strict scientific success, defined as achieving full repair reward or agreement with the private scientific references.

Incomplete or budget-exhausted deliveries are considered unsuccessful.

Evan: Interesting.

They also compare multiple models, right?

Ashley: Exactly.

They compare fifteen models from eight providers, using different harnesses like Codex, Claude Code, and Gemini CLI.

Each model gets the same task-specific container and instructions, with a one-hour episode budget.

This setup allows the authors to compare the model-harness systems under identical task interfaces.

Evan: What were the key findings from these comparisons?

Ashley: Fable 5.1 achieved the highest observed success rate at 67.1%, followed by Opus 5 and Astra at 64.6% and 63.1%, respectively.

Despite these high success rates, the leading models left roughly one third of the tasks unsolved within the given budget.

Evan: That's a significant portion.

What's interesting about how these models allocate their budgets?

Ashley: The paper presents budget-response profiles that show how quickly different models solve scientific tasks.

For example, Astra reaches 49.6% success in the first ten minutes against Fable’s 25.9%, but Fable eventually overtakes Astra around the 31-minute mark.

Evan: That suggests different efficiency dynamics.

How about resource use and cost efficiency?

Ashley: Indeed, the paper notes that more expenditure doesn't guarantee greater success.

For instance, Fable achieves 67.1% success at an estimated $7.90 per task, while Astra achieves 63.1% at $3.56 per task.

Resource use and scientific correctness are distinct dimensions, and profiling helps understand these trade-offs.

Evan: So, what did the authors learn from scientific interaction trajectories, and how did they use this data?

Ashley: The authors record scientific interaction trajectories, including how models inspect code, use tools, and respond to execution feedback.

They found that learning from these demonstrations improves scientific-code repair and transfers positively to public benchmarks.

Evan: And how is this improvement measured?

Ashley: They fine-tuned models like Qwen3.5-4B, Qwen3.5-9B, and Qwen2.5-72B-Instruct on verified ScienceIDE demonstrations.

Each final checkpoint demonstrated higher scientific repair rewards across various scientific environments, alongside gains on public benchmarks in code, reasoning, and knowledge tasks.

Evan: That seems promising.

Do these benchmarks cover a wide range of evaluations?

Ashley: Yes, they cover scientific knowledge, mathematical reasoning, code understanding, repair, and generation.

Within each model pair, both checkpoints are evaluated under identical conditions, showing considerable improvements in various tasks.

Evan: What about reinforcement learning?

How does it tap into this environment?

Ashley: The scientific environments also provide verifier feedback directly as RL rewards.

For instance, they tested Qwen3.5-4B using two environments: LAPS and MITgcm-biogeo.

The RL feedback substantially improved the held-out scientific reward, with notable increases in training reward and reductions in budget truncation.

Evan: That sounds like a robust improvement strategy.

This approach seems quite effective for long-horizon scientific tasks.

Ashley: The paper reports concurrent gains in reward and reductions in truncation, supporting verifier-guided scientific repair.

This translates into higher training and held-out rewards and less truncation, which ultimately benefits the policy optimization.

Evan: Thanks, Ashley.

So that concludes our detailed coverage of the Experiment and Results section.

Evan: Okay, Ashley.

Let’s move on to the Related Work section of the paper.

How does this research fit into the broader landscape of AI and scientific inquiry?

Ashley: This paper bridges two complementary directions in AI and science.

The first is AI for Science, where models and agents are used for scientific prediction, computation, and discovery.

The second is Science for AI, which draws on scientific ideas, data, and environments to develop AI capabilities.

Evan: That makes sense.

Can you describe some examples of AI for Science mentioned in the paper?

Ashley: Sure.

Among the specialist scientific models, they mention protein structure prediction, PDE learning through neural operators, and weather forecasting.

Each of these models develops domain-specific capabilities by leveraging specialized data and objectives.

Evan: And what about scientific discovery through agents?

Any interesting advancements there?

Ashley: Yes, definitely.

For mathematics, physics, and biology, we see advancements like Claude’s computer-checked Fermat formalization and GPT-5.2’s gluon-amplitude conjecture.

There are also reports on GPT-5's contributions to an Erdős problem and an immune-cell mechanism.

AlphaEvolve further extends program-search approaches to mathematical discovery.

Evan: What about applications beyond these specific cases?

Ashley: There are broader AI frameworks aimed at autonomous scientific research.

Examples include biological hypotheses generation, like Robin’s ripasudil proposal for macular degeneration, which was tested experimentally.

Kosmos has reported on broader scientific discoveries as well.

Evan: It seems like scientific agents are making significant strides.

How does this paper position itself among these efforts?

Ashley: The key difference here is the creation of reusable execution and verification interfaces.

This allows the Scientific cases handled by their environments to be used across different tasks and models.

The emphasis is on making scientific workflows not just executable but also verifiable and reusable for learning purposes.

Evan: What about evaluating these scientific capabilities?

How does this paper approach that?

Ashley: Evaluation settings differ significantly.

Some notable mentions are SciCode, ScienceAgentBench, and LAB-Bench, which are all curated problem sets meant to measure scientific capabilities.

There are also broader scientific benchmarks like CORE-Bench and PaperBench that focus on the reproduction of existing results.

Evan: Interesting.

Do they mention any specific benchmarks related to software tasks?

Ashley: Yes, one closely related benchmark is AInsteinBench, which sources maintainer pull requests from scientific repositories, focusing on test suite structure and difficulty annotation.

Another is SWE-bench Science, which highlights how software tests alone can under-specify scientific requirements.

Evan: So how does ScienceIDE distinguish itself from these benchmarks?

Ashley: ScienceIDE focuses on creating an ecosystem where scientific modules are defined, validated, and reused across different tasks and models.

It transforms expert verification into executable environments that can be scaled and adapted for various applications.

This is a step beyond static benchmarks, offering dynamic interaction and feedback.

Evan: That sounds like a powerful framework.

How does Science for AI fit into all of this?

Ashley: Science for AI involves integrating scientific data, concepts, and evaluations into AI development.

Statistical mechanics, for example, have inspired energy-based models, and various scientific datasets have supported the pretraining of language models.

These contribute to creating environments where AI can learn from scientific interactions.

Evan: What about interactive learning environments?

Ashley: Interactive learning environments separate environments, actions, and optimization processes.

There are several precedents in game-based, procedural, embodied, and web-based settings.

Language-model agent suites like SWE-Gym and R2E-Gym provide verifiable tasks and agent supervision through interactive coding and dynamic benchmarks.

These environments extend to formal mathematics, where proof-checker feedback simulates learning from experience.

Evan: So it’s about bringing these principles into the scientific domain?

Ashley: Exactly.

Making scientific experience reusable involves the same concepts—defining clear, verifiable interactions that translate into meaningful learning episodes.

The ScienceIDE approach tailors these mechanisms specifically for scientific tasks.

Evan: Thanks, Ashley.

That was very thorough.

So there you have it, folks, the Related Work section of the paper.

Evan: Alright, Ashley, we've explored the Introduction, Methods, Experiments, and Related Work sections of this paper.

Let’s wrap up by summarizing the key contributions and takeaways.

Ashley: Sure, Evan.

One of the main contributions of ScienceIDE is that it turns scientific codebases into programmable environments that can be used for agent learning.

By encapsulating scientific cases and acceptance criteria defined by experts, it creates reusable and executable environments.

Evan: And this addresses the scientific experience bottleneck, right?

Ashley: Exactly.

This bottleneck exists because scientific codes often come with complex toolchains and configurations, making reproducibility difficult.

By turning these codes into structured environments, ScienceIDE aims to overcome these challenges.

Evan: Another key point is the integration of these environments into supervised fine-tuning and reinforcement learning setups, which enhance model capabilities through scientifically validated interactions.

Ashley: That's right.

The experimental results show that models adapted from ScienceIDE environments perform better in scientific-code repair tasks and general-purpose benchmarks.

This indicates positive transfer from scientific experience to broader AI capabilities.

Evan: This comprehensive approach not only helps in model learning but also aims to make scientific software a shared substrate for developing scientific intelligence.

It creates an integrated workspace where both agent learning and scientific practice can coexist.

Ashley: The next steps for ScienceIDE include broadening scientific coverage and testing transferability across different codebases and task families.

Open-ended research aims to accommodate more complex scientific hypotheses and outcomes.

Evan: Thanks for that summary, Ashley.

And thank you to our listeners for joining us today on this deep dive into ScienceIDE.

If you found this episode insightful, do check out the original paper for more detailed findings.

We hope you enjoyed today's episode.

Ashley: We’ll be back with more fascinating papers from the Hugging Face daily list.

Don’t forget to subscribe and join us every day for your dose of AI research.

See you next time!