Actor, Judge, or Teacher? Understanding Decision Models in Long-Horizon Agents

Evan: Welcome to the Daily Paper Cast, your source for cutting-edge research.

Ashley: Today's paper from Hugging Face's daily paper list dated November 13, 2026, with 22 upvotes, is titled 'Actor, Judge, or Teacher?

Understanding Decision Models in Long-Horizon Agents.'

Evan: The first two authors are Haochen Liu and Qianhao Zhu, and the corresponding author is Yongliang Shen from Zhejiang University.

Ashley: Decision models answer a question with a typed output, such as a choice among options with a probability for each.

These models can either choose an agent’s actions, evaluate actions already taken, or score every action to teach a policy via reinforcement learning.

Evan: Previous research mainly evaluated decision models based on their performance as actors, focusing on how well they choose actions.

However, this study investigates if a decision model that acts poorly can still effectively teach a policy.

Ashley: In this paper, the authors compare three dedicated decision models and three general large language models, LLMs, as actors, judges, and teachers on two environments: ALFWorld and WebShop.

Evan: Interestingly, as actors, all six models often fail by being diverted away from a necessary subgoal by an easier one.

But as judges, given the entire trajectory, these models credit steps toward the required subgoal.

Ashley: This study highlights an important distinction: judging whether each action helped achieve the goal, rather than predicting the task's overall success, helps differentiate between useful and useless actions within a trajectory.

Evan: Using JEV, one of the decision models, as a teacher to score actions after seeing the whole trajectory, significantly improved the success rates of trained policies on both ALFWorld and WebShop.

Ashley: Specifically, JEV was found to raise a Qwen2.5-1.5B policy's success on ALFWorld from 72.8% to 82.9%, and on WebShop from 56.8% to 71.4%.

These results suggest that decision models that act poorly can still teach well by leveraging post-episode evaluations.

Evan: The findings imply that decision models' suitability as supervisors should be assessed based on their ability to judge recorded trajectories rather than only their acting performance.

Ashley: And importantly, step-level supervision can be efficiently obtained by querying a frozen decision model after the episode ends, which reduces the need for costly step labels or learned critics during reinforcement learning training.

Evan: This concludes the Introduction section of 'Actor, Judge, or Teacher?

Understanding Decision Models in Long-Horizon Agents.' Stay tuned for more insights from this paper.

Evan: Moving on to the Methods section, Ashley, how did the authors set up their experiment to compare decision models as actors, judges, and teachers?

Ashley: Sure, Evan.

The authors evaluated three dedicated decision models and three general large language models, or LLMs, across two distinct environments: ALFWorld and WebShop.

ALFWorld is a text-based household environment where an agent completes goals by issuing text commands.

For example, the task might be to put a clean sponge on a cart, requiring text commands like 'go to,' 'take,' and 'clean.'

Evan: Hmm, and what about WebShop?

Ashley: WebShop is a shopping environment where an agent searches through a product catalogue, opens product pages, selects options such as size and color, and makes a purchase.

Each instruction specifies product attributes, options, and a price limit, and the task succeeds only when the purchased item matches the given criteria exactly.

Evan: Got it.

So, what exactly were the roles these models were tested in?

Ashley: The authors evaluated each decision model in three distinct roles: Actor, Judge, and Teacher.

As actors, the models chose each action before seeing its effect, effectively predicting the next step to move closer to the goal.

As judges, they evaluated actions in recorded trajectories.

Here, the model could see either the trajectory up to the action, called 'Prefix,' or the whole trajectory, called 'Hindsight.' Each action was judged based on whether it helped achieve the goal, termed 'Progress,' or whether the task would succeed, termed 'Value.'

Evan: And how were they used as teachers?

Ashley: As teachers, the decision models applied the same queries used during judging to score actions in the training trajectories of a reinforcement learning policy.

These scores were added to the outcome advantage signals of GRPO, which stands for Group-based Policy Optimization, to provide step-level supervision.

Evan: That’s quite detailed.

How did they adapt the models for the decision-making interface?

Ashley: Dedicated decision models like JEV, Decider v1.2, and GPT Decisions were queried through their specific decision APIs.

For general LLMs, the base models were Qwen3.8-27B and GPT-6 Luna.

These were adapted by instructing the models to output their decisions and probabilities in JSON, or by using constrained readouts where the first-token probabilities were returned for valid answer labels.

Evan: What about their benchmarks?

How were the tasks structured across the two environments?

Ashley: In ALFWorld, tasks fell into three groups based on the relationships among their requirements.

Pick and Look tasks involved a single object, while Clean, Heat, and Cool tasks required state changes and placements that depended on the object’s state.

For example, heating a mug before placing it on a table.

Tasks like Pick2 had two independent requirements without a set ordering.

Evan: And in WebShop?

Ashley: In WebShop, each task required attribute selection before purchase.

The end result was judged by how well the purchased item matched the original instruction.

The tasks in this environment simulated real-world shopping scenarios.

Evan: Right.

So, how did they ensure that these models provided insightful evaluations both as judges and teachers?

Ashley: The authors compared whether seeing the whole trajectory, rather than just the prefix leading to an action, could better reveal useful actions.

They determined that judging whether each action helped achieve the goal, rather than just whether the task would eventually succeed, produced more accurate evaluations.

For instance, in ALFWorld, the Progress question with Hindsight context raised success rates from 72.8% to 82.9% when training a Qwen2.5-1.5B policy.

Evan: Interesting.

How did the models perform in their role as actors?

Ashley: Interestingly, all model actors struggled with tasks where subgoals needed to be achieved in a specific order.

For example, JEV completed 70.8% of tasks with independent subgoals but only 14.7% of ordered subgoals like heating a mug before placing it on a table.

When these models encountered an easier subgoal, they frequently chose the easier one first, failing to complete the task correctly.

Evan: So, what implications does this have for training models effectively?

Ashley: The key finding is that decision models that act poorly can still teach well, provided their evaluations occur after the entire trajectory is visible.

This ensures that their judgments consider the consequences of actions, which an actor mode does not reveal.

Hence, leveraging these evaluations in reinforcement learning can significantly boost training efficacy.

Evan: All in all, it seems like the choice of context and question is crucial for evaluating the utility of decision models in different roles.

Ashley: Exactly, Evan.

This dual approach of evaluating both the immediate and overall impacts of actions offers a more granular understanding of how actions contribute to reaching the goal.

This allows us to refine how we train and deploy agents for complex tasks.

Evan: So, that finishes the Methods section of the paper, ‘Actor, Judge, or Teacher?

Understanding Decision Models in Long-Horizon Agents.’ It’s fascinating how nuanced evaluating these models can be.

Evan: Alright Ashley, let’s dive into the Experiment and Results section.

How did the experiments play out?

Ashley: Sure, Evan.

The experiments were designed to test the models in their roles as actors, judges, and teachers across both ALFWorld and WebShop environments.

The focus was on understanding how each model performed under different conditions and how these performances influenced training outcomes.

Evan: Interesting.

So, how did the models perform as actors in ALFWorld?

Ashley: In ALFWorld, the models encountered tasks with dependencies and independent subgoals.

When it came to tasks with dependencies, such as heating a mug before placing it on a table, all models struggled significantly.

Evan: Can you give us some figures to illustrate that?

Ashley: Of course.

For instance, JEV completed 70.8% of tasks with independent subgoals, like putting two pencils in a drawer, but only 14.7% of tasks with ordered subgoals.

Decider v1.2, which was the strongest actor overall, succeeded in only 18.8% of Heat tasks.

Evan: That sounds challenging.

What about the models' performance as judges?

Ashley: When the models were used as judges, they evaluated actions in recorded trajectories under either 'Prefix,' where they saw the trajectory up to the action, or 'Hindsight,' where they saw the whole trajectory.

The key metrics were whether each action helped achieve the goal, termed 'Progress,' or whether the task would succeed, termed 'Value.'

Evan: Which approach was more effective?

Ashley: Seeing the whole trajectory and asking whether each action helped achieve the goal marked a significant improvement.

For example, under Hindsight context, JEV’s mean contribution score increased from 0.41 to 0.59.

All six models showed this pattern and provided more accurate evaluations when observing the full trajectory.

Evan: So, seeing the whole trajectory is really beneficial.

Does this translate into better training outcomes?

Ashley: The Hindsight context with Progress scores resulted in the most effective training outcomes.

Combining these scores with outcome rewards during training significantly boosted policy performance.

For example, the Qwen2.5-1.5B policy’s success rate increased from 72.8% to 82.9% on ALFWorld using Progress scores with Hindsight context.

Evan: That's quite an improvement.

How did they ensure the cost-effectiveness of this approach?

Ashley: The authors evaluated the cost by querying JEV through OpenRouter to score each executed action during training.

The training run costs ranged from 22 to 30 USD depending on the specific experiment setup, demonstrating that effective step-level supervision can be obtained at a reasonable price.

Evan: Impressive.

Did the models perform similarly in the WebShop environment?

Ashley: Yes, the results were consistent in WebShop as well.

The Qwen2.5-1.5B policy trained with Hindsight and Progress raised success rates from 56.8% to 71.4%.

The models generally provided more accurate evaluations and improved policy performance in both environments when the full trajectory was visible.

Evan: So the take-home message is clear: models that might struggle with acting tasks can still teach effectively when they evaluate actions post-episode.

This duality is quite fascinating.

Ashley: Exactly.

And this approach leverages the strengths of decision models to guide policy training, highlighting the importance of the context and the questions asked during evaluations.

Evan: Alright, that wraps up the Experiment and Results section of the paper ‘Actor, Judge, or Teacher?

Understanding Decision Models in Long-Horizon Agents.’

Evan: Ashley, let’s delve into the Related Work section.

What existing research did the authors consider in this study?

Ashley: Definitely.

The paper synthesizes knowledge from three main areas: decision models, LLM judges and process supervision, and credit assignment for LLM agents.

Let's start with decision models.

Evan: Sure.

So, what are decision models exactly?

Ashley: Decision models, like JEV, GPT Decisions, and Decider v1.2, respond to queries by choosing among given options or assigning scores.

Historically, these models have been mainly evaluated by their actions.

For instance, Yang et al.

studied JEV’s action selection and found improvements when using explicit subgoals and simulated action outcomes.

Similarly, JevAgentHarness trained decision models with correct-option labels and JEV’s probabilities, allowing them to share control with an LLM.

This research goes further by evaluating decision models not just as actors but also as judges and teachers.

Evan: Interesting.

What did their investigation reveal about decision models' role in evaluations?

Ashley: Yes, the paper showed that performance as actors does not necessarily reflect the models' value as judges and teachers.

Decision models must be evaluated in terms of how they score recorded trajectories, which is a nuanced approach that differs from merely assessing their action selection accuracy.

Evan: That makes sense.

Now, what about LLM judges and process supervision?

Ashley: LLM judges provide evaluation capabilities without task-specific training and are commonly used to measure the quality of final responses.

For instance, Zheng et al.

introduced LLM-as-a-judge, while Bai et al.

developed methods for AI feedback to improve harmlessness.

Lightman et al.

suggested process reward models for intermediate steps, which demand step-level labels for each domain.

The current study builds on these ideas by showing that decision models can score actions without needing step labels, underscoring the importance of the question being asked and the context shown.

Evan: Right, so the context and questions are critical there.

How about credit assignment in LLM agents?

Ashley: Credit assignment in LLM agents focuses on long interactions with their environment.

Traditional methods like GRPO, Group-based Policy Optimization, align all steps' advantages within a trajectory.

Recent enhancements offer finer credit signals structured from rollouts, such as GiGPO, which compares actions across trajectories.

Hindsight credit assignment, as proposed by Harutyunyan et al., uses subsequent events to evaluate previous actions.

This paper's method parallels these by leveraging step-level judgments from a frozen decision model to train policies more effectively.

Evan: That's impressive.

So, they integrate hindsight evaluation into reinforcement learning using decision models.

Ashley: Exactly.

Additionally, this work resonates with the weak-to-strong generalization concept presented by Burns et al.

Here, a strong model, trained on labels from a weaker one, can outperform its teacher.

Similarly, a fixed decision model judged trajectories post-episode, guiding the policy learning beyond its acting limitations.

Evan: Clearly, this research stands on the shoulders of significant prior work, yet brings innovative insights too.

Ashley: Precisely.

It merges established methods with novel evaluations, showing how decision models can transition from inefficient actors to effective teachers through post-episode judgment strategies.

Evan: Fascinating synthesis of previous research.

That concludes the Related Work section.

Evan: Ashley, let's wrap up by summarizing the key contributions and takeaways from the paper.

Ashley: Sure, Evan.

This paper, 'Actor, Judge, or Teacher?

Understanding Decision Models in Long-Horizon Agents,' presented a systematic study of decision models in three roles: Actor, Judge, and Teacher.

Evan: So, what were the major findings of the study?

Ashley: First, the authors found that decision models often struggle as actors when tasks involve dependent subgoals.

Models like JEV, although capable of completing tasks with independent subgoals, frequently failed on tasks requiring a specific order, such as heating a mug before placing it on a table.

Evan: That's quite a limitation.

How do these models perform better as judges?

Ashley: As judges, seeing the whole trajectory allowed the models to better evaluate each action's contribution to achieving the goal.

This approach revealed that post-episode evaluations enabled finer distinctions between useful and useless actions within a trajectory.

Evan: And what was discovered about their role as teachers?

Ashley: In their role as teachers, using these models to score actions after seeing the full trajectory significantly improved training outcomes.

For instance, JEV as a teacher raised the success rates of the Qwen2.5-1.5B policy on ALFWorld from 72.8% to 82.9%, and on WebShop from 56.8% to 71.4%.

This highlights that models that act poorly can still teach effectively when their evaluations consider the consequences of actions post-episode.

Evan: So the takeaway is clear: For effective agent training, decision models should be evaluated based on their ability to judge recorded trajectories, not just their acting performance.

Ashley: This study underscores the importance of context and the specific questions asked during evaluations.

Leveraging these elements in reinforcement learning can help transform decision models from subpar actors into stellar teachers.

Evan: That wraps up today's episode of the Daily Paper Cast.

We hope you found these insights on decision models in long-horizon agents as fascinating as we did.

Ashley: Make sure to join us next time for more explorations of cutting-edge research from the world of AI, NLP, CV, and beyond.

Evan: And don't forget to subscribe to our podcast and share it with your colleagues and friends.

Until next time, stay curious and keep learning!

Ashley: Goodbye everyone!