SO-Arena: Benchmarking the Capability Spectrum and Reliability of System One Models

Evan: Welcome to Daily Paper Cast.

Ashley: Today's paper is from the Hugging Face daily paper list of November 13, 2026, with 40 upvotes.

Evan: We're discussing the paper titled 'SO-Arena: Benchmarking the Capability Spectrum and Reliability of System One Models'.

Ashley: It's authored by Lichen Li, Ziping Dong, with corresponding author Xinchao Wang, all from the National University of Singapore.

Evan: Ashley, can you unpack what System One models are and why they matter in decision-making?

Ashley: Sure, System One models are designed for fast, general-purpose decision-making without the need for lengthy token generation.

They predict a distribution over predefined outcomes in a single forward pass, which reduces inference latency significantly compared to models that generate reasoning tokens.

Evan: That sounds efficient, but where's the catch in understanding their capabilities and reliability?

Ashley: Precisely, Evan.

While these models are promising for quick decisions, understanding their full spectrum of capabilities and the reliability of their decisions requires more than just aggregate performance on conventional benchmarks.

Conventional evaluations give a limited view and don't always reflect real-world application complexities.

Evan: So how does SO-Arena address these limitations?

Ashley: The SO-Arena benchmark introduces a more comprehensive evaluation framework.

It characterizes the capability spectrum and decision reliability of System One models across eleven domains and five capability families.

Evan: What exactly does this benchmark include in terms of capabilities?

Ashley: For capabilities, SO-Arena offers evidence-grounded evaluations through verified labels and reliable sources.

It includes fine-grained assessments of model performance based on domain-specific difficulty criteria and realistic interactive evaluations where models execute actions iteratively in sandbox environments.

Evan: But how does it ensure the reliability of these models' decisions?

Ashley: Great question.

Beyond accuracy, SO-Arena assesses decision reliability through two complementary measures: decision stability under controlled changes to the option set and the ability to recognize when no valid option is available.

This gives a more nuanced understanding of how consistent and trustworthy these models are, especially when options are reordered or missing correct answers.

Evan: Interesting.

How extensive is this benchmark?

Ashley: SO-Arena includes evaluations across 21 models, revealing substantial differences in how scaling and reasoning affect performance across different domains.

The benchmark spans 2,919 tasks organized into five capability families: semantic understanding, knowledge and reasoning, safety and policy compliance, evidence assessment, and agent task completion.

Evan: What are some key findings from these evaluations?

Ashley: One key finding is that strong overall performance does not necessarily imply reliable decision-making.

For instance, the best-performing System One model ranked only seventh in terms of reliability.

This highlights the importance of evaluating both capability and reliability as complementary dimensions.

In other words, while some models excel in capability, their decision reliability may still lag behind.

Evan: So, SO-Arena provides a systematic view of the strengths and limitations of System One models.

What are the main contributions of this paper?

Ashley: The paper's main contributions include: introducing a comprehensive benchmark, establishing a traceable evaluation process with manually verified labels, using domain-specific difficulty tiers to assess model capabilities, and developing separate reliability measures.

These insights collectively aim to advance both the evaluation of model capabilities and the reliability of their decisions.

Evan: And that concludes the introduction section of the paper.

Stay tuned as we delve into the methods and experiments behind SO-Arena in the next segment.

Evan: Alright, let's dive into the methodology behind SO-Arena.

Ashley: SO-Arena's benchmark is constructed from 2,919 tasks across eleven domains.

These domains are grouped into five capability families: semantic understanding, knowledge and reasoning, safety and policy compliance, evidence assessment, and agent task completion.

Evan: It sounds like a comprehensive setup.

What kind of tasks are included, and how are they organized?

Ashley: The tasks are derived from 40 different public sources, ensuring a wide range of challenges.

The benchmark includes 2,300 static tasks and 619 environment-based tasks.

Static tasks involve judging a fixed input, while environment-based tasks require models to assess or select actions based on observations from web or tool environments.

Evan: How does the paper ensure these tasks are high quality?

Ashley: Quality control is paramount.

Task labels from these sources are manually verified to ensure they are supported by the information available to the models.

This prevents any discrepancies between the task inputs and the expected outputs.

Items with questionable references are dropped after human and large language model reviews.

Evan: Got it.

And what about the structure of these tasks?

How are they formulated for evaluation?

Ashley: Each domain is divided into difficulty tiers, except for two domains: Sentiment & Emotion Understanding and Knowledge & Commonsense.

These tiers are defined relative to each domain and reflect the complexity of tasks.

For instance, in the Math domain, tasks are distinguished based on their calculation steps and source difficulty levels.

Evan: That's a thoughtful approach.

Could you give us examples of how tasks are crafted for specific domains?

Ashley: Let's take the Semantic Understanding capability family.

The Sentiment & Emotion Understanding domain includes tasks from various datasets like Amazon Reviews and GoEmotions; models predict review ratings or classify emotions.

For Intent & Routing, tasks ask models to identify the intent behind user questions or track customer goals within conversations using sources like CLINC150 and BANKING77.

Evan: So it's not just about answering questions but understanding their context and nuances.

What about the evaluation metrics?

Ashley: Evaluation in SO-Arena spans several metrics.

The capability score for each model averages the hardest tier of each domain while giving equal weight to each domain.

This score focuses on performance differences on demanding tasks.

Evan: But how does SO-Arena specifically measure reliability?

Ashley: Reliability is measured through stability under option order changes and recognition of missing correct options.

For option order stability, models are tested to see if they consistently select the same option regardless of how choices are reordered.

For null discrimination, models must flag when no correct options are available, predicting higher probabilities for the absence of any valid answer.

Evan: It sounds like these metrics go beyond traditional accuracy.

How does the benchmark combine these reliability measures?

Ashley: SO-Arena combines these metrics into a reliability score using their geometric mean.

Models score positively only when both stability and null discrimination exceed chance levels.

This ensures a nuanced evaluation where strong performance translates to reliable decision-making consistently.

Evan: And what kind of models were evaluated using SO-Arena?

Ashley: The benchmark evaluates 21 models, including both open-weight models ranging from 0.4 billion to 31 billion parameters, and closed decision APIs like GPT-6 Luna and Sol.

These models span different sizes and configurations, some designed for quick decisions without reasoning and others equipped with reasoning capabilities.

Evan: That's quite a variety.

How are these models run and timed during the benchmark?

Ashley: Each model runs in settings documented by their developers, with performance paired with latency since speed is crucial for System One models.

The evaluations measure how consistent and reliable models are when faced with changes in their available options.

Evan: This methodology seems robust.

Did they find anything surprising about the model performances?

Ashley: Yes, they found that System Two models like GPT-6 Sol and Luna still lead in overall performance but at a higher inference cost.

However, several System One models demonstrated considerable speed advantages, answering within 0.3 seconds per task, which is tens of times faster than the System Two models.

Evan: So, while being faster, how do these System One models fare in reliability?

Ashley: Interestingly, higher performance doesn't always mean higher reliability.

For instance, JevAny-27B and Jev API 1.14 excelled in reliability even though they ranked lower in capability.

This pattern underscores the importance of evaluating both dimensions independently.

Evan: What can we infer from these findings concerning future model development?

Ashley: These findings suggest that advancing model capabilities should go hand in hand with improving their decision reliability.

This approach ensures that models are not just capable but also consistent and trustworthy in real-world applications.

Evan: That concludes our exploration of the methods used in SO-Arena.

Up next, we'll analyze the experiments and results section.

Evan: And now, let's delve into the experiments and results presented in the SO-Arena paper.

Ashley: Evan.

SO-Arena evaluates 21 models in terms of capability, inference latency, and decision reliability.

These models span various sizes, from 0.4 billion to 31 billion parameters, and both open-weight models and closed decision APIs are included.

Evan: Let's start with the overall performance and latency.

What did they find?

Ashley: The paper finds that System Two models, like GPT-6 Sol and Luna, still lead in overall performance.

However, this comes at a much higher inference cost, as these models need to generate additional reasoning tokens before providing answers.

On the other hand, System One models demonstrated significant speed advantages.

Every open-weight System One model answered within 0.3 seconds per task on an H100 GPU, which is tens of times faster than GPT-6 Sol and Luna.

Evan: So speed is where System One models shine.

What about reliability?

Ashley: Reliability is a bit more nuanced.

The paper measures reliability through decision stability under option reordering and null discrimination, the ability to recognize when no valid answer is available.

While performance and reliability are positively correlated, higher performance does not always imply higher reliability.

For instance, JevAny-27B and Jev API 1.14 ranked highest in reliability but were only eighth and sixth in terms of performance.

Evan: What insights did the paper offer about performance across different domains?

Ashley: The paper categorized the domains into three main sets: direct semantic judgment, fine-grained grounded decision, and multi-step derivation.

In direct semantic judgment tasks, System One models performed comparably to System Two models.

In domains requiring fine-grained grounded decisions, models like GLiDE came close to Luna.

However, in multi-step derivation tasks such as math and scientific reasoning, System One models fell significantly behind.

Evan: What factors help narrow the gap in performance between System One and System Two models?

Ashley: The gap narrows most notably with advancements in backbone capabilities and reasoning.

Larger, more advanced backbones improved performance in fine-grained decision tasks.

Activities involving multi-step derivation benefitted most from explicit reasoning processes like chain-of-thought, lifting performance close to Luna, especially in tasks that involve math and symbolic reasoning.

Evan: Did the paper find any interesting patterns between model size and reliability?

Ashley: Yes, size alone doesn't decide reliability.

For option order stability, most models kept their answers consistent regardless of how options were reordered.

However, reliability varied widely in null discrimination, which indicates a model's awareness of when no valid option is present.

Some smaller 4 billion parameter models outperformed larger models like 9 billion and 27 billion models in this aspect.

Evan: That's intriguing.

How consistent are the models' decisions when reordering options?

Ashley: Most models show high order stability; 14 out of 18 scored above 0.8 in this metric.

Models change their answers more frequently when the reorderings involve questions they answer incorrectly compared to those they answer correctly.

This indicates that instability mainly arises where models have lower performance.

Evan: So, reliability in recognizing missing correct options is a bit less predictable?

Ashley: Indeed.

Null discrimination does not follow performance as closely as option order stability.

For example, systems like Kev-27B, Jev API 1.14, and JevAny-27B exceeded the top-performing System One models in null discrimination.

This highlights that a robust performance metric isn't enough to gauge real reliability; measuring it directly is crucial.

Evan: These insights seem crucial for understanding where and how these models can be applied effectively.

Any final thoughts from the paper's experiments section?

Ashley: Overall, the findings stress the necessity to enhance both capability and reliability.

There's a notable divergence between models excelling in general performance and those ranking high in reliability, underscoring the importance of multi-faceted evaluations.

Hence, advancing model capabilities should be coupled with improvements in their decision-making consistency.

Evan: And that concludes our coverage of the experiments section from SO-Arena.

Stay tuned as we'll next discuss the related work section and its connection to these findings.

Evan: Let's now shift our focus to the related work section of the paper and explore how the findings of SO-Arena fit within the broader landscape of model evaluation and decision-making benchmarks.

Ashley: Evan.

This section provides valuable context by situating SO-Arena within the existing body of research on System One models and decision benchmarks.

To start, System One models are not a new concept.

Jev introduced these models as general-purpose tools designed to expose predictions over explicitly specified outcomes, such as classification, routing, scoring, and verification tasks.

Evan: So Jev laid the groundwork for these models.

How have models like Kev and JevAny built on this foundation?

Ashley: Indeed.

Kev and JevAny have expanded on Jev’s foundational work by providing language-model-based implementations.

Research efforts by teams like SII-Decision and Releasing the Inner Jev explored how reasoning-informed predictions can be distilled into a single-pass readout, making decision-making more efficient without compromising accuracy.

These implementations highlight the various ways System One model capabilities can be derived from pretrained models.

Evan: That sounds like a rigorous development path.

What about the evaluation aspect?

Have there been previous benchmarks aimed at evaluating these models?

Ashley: Yes, there have been several efforts.

JevBench is one such benchmark that assesses capability, calibration, latency, and inference cost.

Additionally, JEVal covers ten application domains, offering insights into how local decisions can compose into broader agent workflows and social simulations.

OpenDecisions further includes difficulty tiers and controlled perturbations, providing a nuanced view of model performance.

Evan: So SO-Arena extends these previous benchmarks.

What unique contributions does SO-Arena bring to the table?

Ashley: SO-Arena complements these efforts by introducing within-domain difficulty tiers, controlled history-dependent task verification, and a distinct reliability score.

This broader evaluation framework systematically examines decision competence across a variety of scenarios, ensuring thorough assessment of both capability and reliability.

Evan: Interesting.

How does this benchmark relate to broader capability evaluations like MMLU or HELM?

Ashley: Good question.

While MMLU measures knowledge and problem solving across a range of academic and professional subjects, SO-Arena focuses on generalist competence through decisions over supplied evidence and specified alternatives.

Similarly, BIG-Bench Hard challenges models with demanding tasks and explores the effects of explicit reasoning.

Humanity's Last Exam pushes evaluation toward expert-level questions.

SO-Arena complements these broader benchmarks by emphasizing decision reliability and the ability to maintain judgments under changing informational demands.

Evan: So, it’s about balancing breadth with the depth of assessment.

Does SO-Arena also touch on automated evaluation tools?

Ashley: Yes, it does.

Research on LLM-as-a-judge, as seen in RewardBench and JudgeBench, establishes the role of automated evaluations but also highlights biases like position, verbosity, and self-enhancement.

VerifyBench examines agreement across scientific domains, emphasizing sensitivity to response presentations.

SO-Arena connects these perspectives to general-purpose decision evaluation, assessing decisions based on documents, structured states, or interaction histories, rather than just candidate answer pairs.

Evan: That's quite comprehensive.

What about evaluations in agentic environments, where models interact with web or tool environments?

Ashley: Indeed, agentic evaluations are crucial.

Platforms like Mind2Web evaluate action predictions across various tasks and websites, while WebArena provides reproducible web environments for testing functional capabilities.

τ-bench examines policy-constrained tool interactions, emphasizing reliability across repeated trials.

VerifierBench focuses on generating executable checkers against valid solutions and common errors.

SO-Arena evaluates decision models in these roles by having them choose actions and verify recorded executions, ensuring the models can handle both policy and verification tasks effectively.

Evan: So SO-Arena integrates these various evaluation approaches to measure comprehensive competence.

Are there other benchmarks it relates to?

Ashley: Yes, SO-Arena also builds on works like ReflexBench and System1-Arena, which compare System One decision models across numerous benchmarks.

It expands upon these by offering a more granular view that separates capability and reliability into distinct, complementary dimensions.

Evan: It sounds like a significant step forward in benchmarking models.

Would you say these related works together highlight the evolution of model evaluation frameworks?

Ashley: These integrated approaches underscore the maturation of benchmarking methodologies, focusing on detailed evaluations that go beyond traditional metrics like accuracy alone.

They collectively stress the need for balanced assessments that look at both performance and reliability of models across diversified tasks and environments.

Evan: And that wraps up the discussion on the related work section.

Stay tuned as we proceed to the conclusion and implications of the SO-Arena paper.

Evan: We're reaching the end of today's episode, so let's summarize the key contributions and takeaways from the SO-Arena paper.

Ashley: Certainly.

The SO-Arena benchmark is a significant advancement in understanding System One models.

It provides a detailed characterization of their capability spectrum across eleven domains, organized into five capability families.

Evan: One of the standout contributions is the benchmark's comprehensive approach to evaluating both model capabilities and decision reliability.

Ashley: Exactly.

By combining fine-grained capability assessments with measures of decision stability and null discrimination, SO-Arena goes beyond traditional accuracy metrics.

It ensures models are evaluated for both their competence and their consistency in decision-making.

Evan: Another key takeaway is the performance gap between System One and System Two models.

While System Two models like GPT-6 Sol and Luna lead in overall performance, System One models are much faster, offering significant speed advantages.

Ashley: Yet, reliability varies independently of size.

High performance doesn't always equate to high reliability, which emphasizes the importance of separate evaluation dimensions.

Models like JevAny-27B excelled in reliability despite their lower ranking in overall performance.

Evan: And these insights reinforce the need to focus on advancing both capabilities and reliability in future model development.

Ensuring models are not only capable but consistently reliable in their decision-making is vital for practical applications.

Ashley: The paper's contributions point toward a more nuanced and thorough benchmarking approach, paving the way for future research to build on these findings and improve model evaluation frameworks.

Evan: That wraps up our discussion on SO-Arena.

Thank you for joining us today.

Ashley: We hope you found the insights valuable.

Be sure to tune in for our next episode as we continue to explore the latest in AI research.

Evan: Until next time, stay curious and keep learning.

Goodbye!

Ashley: Goodbye!