StepAudio 3 Realtime Technical Report

Evan: Welcome to Daily Paper Cast.

Ashley: Today, we're discussing a paper from the Hugging Face daily paper list of September 16, 2026, which received 21 upvotes.

Evan: The paper is titled 'StepAudio 3 Realtime Technical Report'.

Ashley: The first two authors are Bin Lin and Bo Zhao, and the corresponding author is Boyang Zhang from the StepFun-Audio Team.

Evan: Let's dive into the introduction of the paper.

Ashley: Natural spoken interaction requires a system to follow the user while managing its own response.

This interaction can be quite complex due to the need to handle pauses, acknowledgments, and interruptions fluidly.

Evan: Right, and this complexity is compounded when dealing with multi-turn requests, where the model must reason carefully while keeping the conversation responsive.

Tool use introduces an additional challenge since tasks initiated by spoken exchanges might outlast the dialogue itself.

Ashley: Advances in speech recognition have leveraged acoustic representations combined with the linguistic knowledge of large language models.

Audio-language models now support broader acoustic understanding and direct speech generation.

Evan: Indeed, streaming and full-duplex systems have become more sophisticated, allowing listening and speaking to overlap.

These capabilities consider linguistic content, vocal delivery, and conversational timing.

Ashley: StepAudio 3 Realtime builds on the StepAudio series’ shared audio-language foundation.

The focus of StepAudio 3 Realtime is on the seamless coordination of perception, reasoning, and action as a conversation unfolds.

Evan: Interesting.

How do they organize these core functions?

Ashley: They organize them into a continuous loop of listening, conversing, thinking, and acting.

Deep Perception is designed to capture linguistic and nonverbal acoustic evidence.

Seamless Duplex utilizes both user and model speech to manage the conversational floor.

Evan: Think-While-Speaking, as mentioned in the abstract, coordinates reasoning with spoken delivery, supported by Adaptive Thinking and multi-token prediction.

How does the voice agent fit into this mix?

Ashley: The Voice Agent extends the interaction to tasks that require tools.

It processes the request and required arguments before execution, then incorporates returned evidence into the ongoing conversation.

This means users can continue speaking while a task progresses in the background.

Evan: That sounds incredibly efficient.

So, what kind of performance has StepAudio 3 Realtime achieved?

Ashley: StepAudio 3 Realtime leads the reported baselines on four out of eight audio-understanding benchmarks and achieves the highest reported Overall score on the Artificial Analysis Full-Duplex Bench.

The results identify some gaps in multi-turn constraint following and retail tool-use tasks, which are discussed in later sections.

Evan: It seems like the introduction sets a strong foundation for understanding the model's capabilities and its real-time interaction potential.

Ashley: This concludes the introduction section of the paper.

Evan: Alright Ashley, let's dive into the methods used in StepAudio 3 Realtime.

Ashley: Evan.

The methods are organized into several key components focusing on real-time spoken interaction.

Evan: That sounds comprehensive.

What's the first major component?

Ashley: The first component is the system architecture.

StepAudio 3 Realtime uses a mixture-of-experts architecture, which consists of approximately 196 billion total parameters with 11 billion active parameters per token.

The language backbone builds on Step 3.7 Flash, while the audio frontend utilizes the Audio Transformer encoder from Qwen3-Omni.

Evan: So, how does the full-duplex input path work within this architecture?

Ashley: The full-duplex input path integrates user and model audio streams.

Audio representations pass through the encoder and adapter to the language model decoder, which also receives text tokens separately.

This setup allows joint conditioning on acoustic and textual context, enabling seamless integration between listening and speaking.

Evan: It's fascinating how they manage such complex interactions.

How about training these models?

Ashley: Training is divided into three stages: modality alignment, multimodal mixed training, and cooldown.

The modality-alignment stage establishes the interface between acoustic representations and the language model.

Multimodal mixed training develops joint audio-text modeling at scale.

Finally, the cooldown stage emphasizes high-quality data to refine the foundation.

Evan: What kind of data are involved in this pretraining process?

Ashley: Pretraining involves a mix of curated audio data processed through an automated pipeline for sound event detection, voice activity detection, metadata assignment, transcription, and language identification.

This ensures high-quality, semantically complete samples suitable for training.

Evan: And after pretraining comes midtraining, right?

Ashley: Exactly.

Midtraining expands context length to accommodate longer dialogue histories and integrates deeper perception, voice-agent interaction, and synthetic conversational data.

This stage significantly increases the proportion of audio-understanding and agent-interaction data.

Evan: You've mentioned Deep Perception earlier.

How does that fit into the fine-tuning process?

Ashley: Deep Perception covers lexical understanding and cues about the speaker, delivery, acoustic events, and temporal structure.

This involves StepAudio 3 ASR Max, which is specialized for transcription.

The ASR-specialized model is fine-tuned with sequence packed examples, applying time-frequency masking for augmentation and producing normalized transcripts.

Evan: Are there any specific techniques to handle rare terminology?

Ashley: Yes, indeed.

Rare names and technical terms are augmented with targeted synthetic examples.

An LLM expands a taxonomy to identify homophones, uncommon characters, abbreviations, and places these terms in natural carrier sentences.

This helps the model differentiate between similar-sounding words.

Evan: And how is audio understanding evaluated?

Ashley: Audio understanding is evaluated through benchmarks like Big Bench Audio, MMSU, MMAU, and more.

These benchmarks help assess audio-grounded reasoning, fine-grained perception, nonverbal acoustic cues, and multi-turn understanding.

Evan: This brings us to Seamless Duplex.

What's its role in conversational floor management?

Ashley: Seamless Duplex is critical for managing conversational timing and reasoning progress.

It handles pauses, backchannels, interruptions, and other speech phenomena.

The system uses a time-interleaved representation, segmenting audio into 320 ms blocks followed by state or text tokens to continually track interaction state.

Evan: So, how does it decide when to listen, speak, or yield the floor?

Ashley: The decision to continue listening, respond, continue speaking, or yield the floor is guided by acoustic evidence and semantic context.

The model interprets overlapping user utterances against its ongoing speech for contextual role determination, distinguishing pauses from turn completions and backchannels from interruptions.

Evan: It seems very nuanced.

How is conversational timing trained?

Ashley: Midtraining adapts the model to full-duplex interaction through supervised training for streaming ASR, voice activity detection, and prediction of utterance completeness.

Post-training refines these behaviors using high-quality interaction data covering turn taking, user backchannels, interruption handling, and background speech rejection.

Evan: And full-duplex interaction needs robust evaluation.

Which benchmarks are used here?

Ashley: StepAudio 3 Realtime ranks first in the Artificial Analysis subset of Full Duplex Bench v1 and v1.5, scoring highly in pause handling, turn taking, user interruption handling, and backchannel handling.

Evan: Now, moving to conversational intelligence and realtime reasoning.

How does StepAudio 3 handle these aspects?

Ashley: StepAudio 3 Realtime combines dialogue and reasoning training with Think-While-Speaking, which coordinates spoken responses with ongoing private reasoning.

Think-While-Speaking uses concurrent calls to the audio model, generating private reasoning while managing streaming output.

Evan: Adaptive Thinking sounds like a key component for reasoning efficiency.

What does it involve?

Ashley: Adaptive Thinking determines whether a response needs explicit reasoning.

It constructs supervision at the assistant-turn level, labeled by whether reasoning changes answer quality.

This helps allocate reasoning effort more efficiently.

Evan: But how is private reasoning accelerated within this framework?

Ashley: Private reasoning is accelerated using Multi-Token Prediction or MTP, which speeds up the decoding process through speculative and multi-token decoding heads.

This reduces sequential target model steps and improves efficiency.

Evan: How effective is Model Merging for capability integration?

Ashley: Model Merging integrates specialized teacher models, leveraging their complementary strengths by parameter averaging.

This balances overall capability without retraining on the unified data mix.

Evan: Now, the full-duplex voice agent brings asynchronous tool execution into play.

How does the model decide when to delegate tasks?

Ashley: Requests are routed based on context: direct responses for stable knowledge, lightweight tools for up-to-date public information, and backend services for complex or prolonged tasks.

The model gathers missing information via user clarification or context retrieval before committing to an external action.

Evan: Can users interact with backend tasks during execution?

Ashley: Users can request updates, provide additional requirements, or shift topics while backend tasks progress.

Returned tool results update the conversational context, guiding subsequent responses and reasoning.

Evan: What does training for conversational tool use involve?

Ashley: Training combines targeted dialogues covering request routing, clarification, private-context retrieval, and execution-time updates with multi-step trajectories focusing on tool-call structure and argument consistency.

Evan: And how is agentic task completion evaluated?

Ashley: Evaluation uses the Artificial Analysis implementation of τ-Voice to assess tool-grounded customer-service tasks under challenging conditions.

This measures the model's ability to track requests and coordinate interaction with tool use.

Evan: Alright, that wraps up the methods section.

It's fascinating how all these components integrate to handle real-time spoken interactions so effectively.

Ashley: Indeed, it's the end of the Method section of the paper.

Evan: We've covered the introduction and methods.

Now, let's move on to the experiments and results.

Ashley: Evan.

StepAudio 3 Realtime underwent extensive evaluation across several key domains: speech recognition, audio understanding, dialogue and reasoning, full-duplex interaction, agentic task completion, and general text capabilities.

Evan: That's a wide array of benchmarks.

How did the model perform in speech recognition?

Ashley: StepAudio 3 ASR Max, the specialized transcription model, showed excellent performance on multiple ASR benchmarks.

It achieved a word error rate of 1.18% on LibriSpeech clean and 2.28% on LibriSpeech other.

It also led AISHELL-1 with a character error rate of 0.49%.

These results highlight its robust transcription accuracy.

Evan: Impressive.

How about its audio understanding performance?

Ashley: StepAudio 3 Realtime excelled in audio understanding tasks.

For instance, it scored 90.6 on the MMSU benchmark, which assesses multi-task spoken language understanding and reasoning.

It also achieved 86.5 on the MMAR benchmark, showing strong performance in deep reasoning across speech, audio, and music domains.

Evan: What other benchmarks were considered for audio understanding?

Ashley: Other benchmarks included AudioMultiChallenge, MMAU, WildSpeech, Step-Caption, and MTalk-Bench.

StepAudio 3 Realtime led four out of these eight benchmarks, with particularly notable gains in areas like MMSU and MMAR.

These results underline the model's robust audio-grounded reasoning, fine-grained perception, and ability to handle nonverbal acoustic cues.

Evan: How did the model perform on dialogue and reasoning benchmarks?

Ashley: On the StepAudioChat dialogue benchmark, StepAudio 3 Realtime reached a macro average of 70.4 in its interactive mode.

It showed competitive results in instruction following, faithfulness, reasoning, memory, knowledge, conversational pragmatics, and persona consistency.

It closely matched dedicated reasoning models while speaking in real-time.

Evan: Could you give an example of a specific dimension within dialogue and reasoning where the model excelled?

Ashley: Of course.

In reasoning, for example, the model scored 73.6, which is a close second to the best-performing model on this dimension, Kimi K3, that scored 81.9.

This indicates strong aggregate reasoning abilities.

Evan: And how did it fare in managing full-duplex interactions?

Ashley: StepAudio 3 Realtime ranked first on the Artificial Analysis subset of Full Duplex Bench v1 and v1.5.

It achieved an overall score of 98.9, with perfect scores for turn taking and user interruption handling.

This demonstrated balanced conversational control, responding appropriately to pauses, interruptions, and backchannels while maintaining conversational flow.

Evan: It sounds like it handled real-time interaction quite well.

What about agentic task completion?

Ashley: For agentic task completion, StepAudio 3 Realtime was assessed using the τ-Voice benchmark, which involves customer-service tasks in domains like airline, retail, and telecom.

It achieved a macro task-success rate of 56.0%, close to the highest score of 56.5 achieved by Grok Voice Think Fast 2.0.

Its performance was particularly strong in the telecom domain, with a score of 70.2%.

Evan: And finally, how did it perform on general text capabilities?

Ashley: StepAudio 3 Realtime also showed excellent results in general text benchmarks.

It achieved an accuracy of 86.8% on the HMMT February 2026 benchmark, which focuses on competition mathematics, and scored 83.0% on GPQA Diamond, a graduate-level question answering benchmark.

These results demonstrate solid performance in text-based reasoning and knowledge tasks.

Evan: It's clear that StepAudio 3 Realtime has robust performance across multiple benchmarks and tasks.

Do these results indicate any areas for improvement?

Ashley: Yes, the results identified some gaps in multi-turn constraint following and retail tool-use tasks.

These are areas where the model has room for further enhancement.

Evan: Thank you, Ashley.

That's the end of the Experiment and Results section.

Evan: Now that we've explored the introduction, methods, and experimental results, let's dive into the related work that influenced the development of StepAudio 3 Realtime.

Ashley: Sure, Evan.

The related work in the paper primarily revolves around notable advances in speech recognition, audio-language models, and full-duplex systems, as well as methodologies incorporated into StepAudio 3 Realtime.

Evan: What are some foundational works cited in this research?

Ashley: The paper references important contributions to robust speech recognition using large-scale weak supervision, such as the work by Radford et al., 2023.

This research integrates acoustic representations with the linguistic knowledge of large language models.

Evan: That sounds crucial.

Were there any specific models or techniques highlighted?

Ashley: Yes.

The work by Bai et al., 2024, on understanding diverse speech and contexts with LLM-based speech recognition is particularly noteworthy.

Lin et al., 2026, further explored multi-token prediction for fast and long-context speech recognition, which laid the groundwork for advances in transcription accuracy in models like StepAudio 3 ASR Max.

Evan: Speaking of advances, audio-language models have seen significant progress.

Could you elaborate on how these models contributed?

Ashley: Certainly.

Borsos et al., 2023, introduced AudioLM, a language modeling approach to audio generation that supports broader acoustic understanding and direct speech generation.

Models like Salmonn and Paralinguistics-aware LLMs have demonstrated the capabilities for natural and nuanced spoken exchanges.

Evan: And full-duplex systems seem to play a vital role in managing conversational dynamics.

What examples can you share from this realm?

Ashley: Full-duplex models like the ones discussed by Wang et al., 2024, and Defossez et al., 2024, are integral to streaming audio systems.

These models enable concurrent listening and speaking, thereby allowing for fluid and responsive dialogue management.

Wu et al., 2026, highlighted chronological thinking in full-duplex spoken dialogue, which plays a crucial part in StepAudio 3 Realtime's Seamless Duplex component.

Evan: It's fascinating how these contributions integrate.

What methodologies were particularly influential in the development of StepAudio 3 Realtime?

Ashley: Methodologies like Mind-Paced Speaking, by Wu et al., 2025, influenced the Think-While-Speaking capability.

It employs a dual-brain approach, with parallel processes for generating private reasoning and managing streaming output.

This methodology supports fluent conversational responses without compromising reasoning depth.

Evan: Were there any specific training techniques used for enhancing these models?

Ashley: Yes, SpecAugment introduced by Park et al., 2019, for data augmentation in automatic speech recognition played a significant role.

Additionally, Recognizer Output Voting Error Reduction (ROVER) by Fiscus, 1997, is utilized to improve transcription accuracy by fusing hypotheses from multiple recognition systems.

Evan: It's remarkable how these methodologies converge to enhance the StepAudio 3 Realtime model.

Let's touch upon the ways these advances impact real-time interaction and tool use in the system.

Ashley: Tool-grounded language modeling approaches like Toolformer by Schick et al., 2023, and Gorilla by Patil et al., 2024, have been significant.

These models demonstrate how language models can effectively integrate external tools for deeper interaction and accurate execution of tasks, which is essential for StepAudio 3 Realtime's Voice Agent.

Evan: Indeed, integrating tools seamlessly into dialogue systems is crucial.

How does this relate to end-to-end task completion benchmarks?

Ashley: End-to-end task completion benchmarks, such as those defined in τ-Voice by Artificial Analysis, 2026, assess full-duplex spoken interaction under challenging conditions.

These benchmarks ensure that models like StepAudio 3 Realtime can track evolving user requests and coordinate spoken interaction with tool use effectively.

Evan: It's impressive how all these elements come together to support robust, real-time spoken interactions in StepAudio 3 Realtime.

Ashley: This concludes the Related Work section.

Evan: We've walked through the introduction, methods, experimental results, and related work for StepAudio 3 Realtime.

Now, let's summarize the key contributions and takeaways.

Ashley: StepAudio 3 Realtime presents a comprehensive approach to real-time spoken interaction.

It coordinates perception, reasoning, and action through a continuous listen-converse-think-act loop.

Evan: One standout feature is Deep Perception, which captures rich acoustic cues to interpret user intent, enabling nuanced spoken exchanges.

Ashley: Precisely.

Seamless Duplex manages synchronized audio streams to fluidly handle pauses, backchannels, and interruptions.

This ensures balanced conversational control.

Evan: And Think-While-Speaking is key to reducing latency while maintaining deep reasoning.

It allows the model to deliberate while producing spoken output in real time.

Ashley: Indeed.

The integration of a Voice Agent extends interactions to tasks requiring tools and asynchronous execution, making the system versatile and efficient.

Evan: In terms of performance, StepAudio 3 Realtime leads in various benchmarks, including audio understanding, full-duplex interaction, dialogue, and agentic task completion.

Particularly notable are its scores on the MMSU, MMAR, and τ-Voice benchmarks.

Ashley: These achievements highlight the model's robust ability to handle multi-turn interactions, complex requests, and dynamic tool use.

While there are areas for improvement, such as multi-turn constraint following and retail tool-use tasks, the overall performance is impressive.

Evan: StepAudio 3 Realtime exemplifies advancements in real-time spoken interaction, showing that seamless coordination between listening, reasoning, and acting is achievable.

Ashley: That's right, Evan.

It's exciting to see how far we've come with audio-language models and their potential to enhance real-time dialogues and interactions.

Evan: Thank you for tuning in to Daily Paper Cast.

We hope you found today's discussion insightful.

Ashley: Be sure to join us for future episodes where we'll continue to explore the latest in AI research.

Until next time, take care!