DexForesight: Grounding Dexterous Policies with Implicit Spatial Foresight

Evan: Welcome to Daily Paper Cast.

Ashley: Today's paper is from the Hugging Face daily paper list of November 11, 2026, and it has received 23 upvotes.

Evan: The title of this paper is 'DexForesight: Grounding Dexterous Policies with Implicit Spatial Foresight.'

Ashley: The first two authors of the paper are Zhongxi Chen and Yi Yang, with correspondence to Zhijie Deng from Sharpa Robotics, SJTU-Sharpa Joint Lab for Embodied AI at Shanghai Jiao Tong University, SJTU.

Evan: Ashley, let's dive into the introduction.

What's the background and main objective behind this work?

Ashley: Sure, Evan.

Dexterous manipulation, which involves controlling a robotic hand with precision, requires rich spatial foresight over both the surrounding scene and the robot hand.

This kind of manipulation must consider how fingers move relative to an object and adapt to changes in the scene, often with parts of the hand occluded from the camera.

Evan: That makes sense.

So, traditional models face challenges in predicting these complex dynamics.

Can you explain how DexForesight addresses this?

Ashley: DexForesight introduces a unique approach to learning spatial foresight by implicitly injecting dynamics-aware world representations into the latent space of a dexterous policy.

Instead of relying on explicit future prediction, it learns an action-conditioned spatial world model from offline scene depth, kinematic hand geometry, and demonstration trajectories.

Evan: Interesting.

How does the world model work during training and deployment?

Ashley: During training, the world model predicts 3D hand-point motion and visual feature changes.

It captures both articulated hand dynamics and scene changes under demonstrated actions, including the motion of fingers that are occluded from the cameras.

Once trained, this model is frozen, and its predictive features are transferred into policy latents.

Evan: And what happens during deployment?

Does the policy still use all those inputs?

Ashley: Not quite.

At deployment, the policy only operates on RGB observations, language, and robot state, without needing depth input or explicit future prediction.

This simplicity and efficiency in deployment are key advantages of DexForesight.

Evan: Got it.

So they managed to retain the complex predictive knowledge without the additional computational overhead during deployment.

That's clever.

What about the main contributions of this paper?

Ashley: The paper makes three major contributions: first, it introduces DexForesight, which aligns dexterous policy latents with predictive representations from a frozen action-conditioned spatial world model.

Second, DexForesight develops this world model using approximately 370 hours of demonstrations across 60 tasks and five categories, combining offline scene depth with kinematic hand geometry to predict 3D hand motion and visual dynamics.

Third, the approach is evaluated on 11 DexJoCo simulation tasks and three real-robot tasks, showing a significant improvement in success rate over the strongest baselines.

Evan: Those numbers sound impressive.

How much improvement are we talking about here?

Ashley: On the DexJoCo simulation tasks, DexForesight improved the overall success rate of the base policy from 49.4% to 60.5%.

For the real-robot dexterous manipulation tasks, DexForesight achieved a success rate of 58.3%, compared to 50.0% for the strongest baseline.

Evan: That's quite an improvement.

It seems like their approach to leveraging spatial foresight really paid off.

Ashley: Indeed, Evan.

By grounding dexterous policies in these predictive representations, DexForesight addresses the complexity of real-world dexterous manipulation.

Evan: Well, that's the end of the Introduction section.

Thanks for breaking it down so clearly, Ashley.

Ashley: My pleasure, Evan.

Let's move on to the next part to discover more about their methodology.

Evan: Now that we have a good understanding of the background and objectives of DexForesight, let's delve into the methodology.

Ashley: Sure, Evan.

The methodology section of this paper provides a detailed overview of how DexForesight uses offline scene depth and hand geometry computed from recorded robot states to train a spatial world model.

Evan: What exactly is the spatial world model, and how does it work?

Ashley: The spatial world model encodes scene observations and hand geometry into a backbone which is initialized from a pretrained model called RynnBrain.

Depth information enters through intermediates fusion blocks, and world queries are used to read the hidden states, conditioned through an action-conditioned adapter.

Evan: Okay, let's break that down a bit.

What kind of data does the world model use during training?

Ashley: The training data includes multi-view RGB images, language instructions, and robot states.

The spatial world model also receives depth and a short history of hand geometry and robot state.

All these inputs are necessary to learn an action-conditioned spatial world model.

Evan: How does the model predict hand motion and visual changes?

Ashley: The model predicts 3D displacements of points attached to different URDF links of the hand through forward kinematics.

This ensures the motion prediction includes even those parts of the hand that are occluded from the camera views.

Additionally, visual changes are captured using feature residuals extracted from RGB images by a pretrained DINOv3 encoder.

Evan: So it sounds like they're covering both the hidden hand dynamics and the changes we can see in the scene.

How do these predictions help in training the policy?

Ashley: Both hand motion and visual dynamics predictions act as conditioning inputs for policy queries during training.

The model leverages these predictions to learn how different parts of the hand and observed scene evolve under demonstrated controls.

Evan: And what happens to the world model after training?

Ashley: After training, the world model is frozen, and its predictive features are transferred into policy queries through a process called query alignment.

This eliminates the need for online depth reconstruction or future generation during deployment.

Evan: Alright, that clears things up.

But what does deployment look like for the policy?

Ashley: During deployment, the policy operates on current RGB observations, language, and robot state inputs.

It uses the aligned query features to generate actions, without the complexity of the spatial world model operation.

Evan: Interesting approach.

Let's talk a bit about how they organized the training data.

What sort of tasks and categories were included?

Ashley: The training data was quite comprehensive.

It included around 370 hours of robot demonstrations across 60 tasks, divided into five categories: spatial arrangement, assembly and contact, rotation and articulation, deformable objects, and tool use.

This broad range of tasks provided varied examples of hand motion and scene changes.

Evan: Wow, that's a lot of data!

How did they estimate the scene depth for these demonstrations?

Ashley: For real-robot demonstrations, the scene depth was estimated offline from recorded stereo RGB sequences using a method called StereoAnyVideo.

For simulations, depth was rendered directly from MuJoCo rollouts.

Evan: What about the training process?

Were there different stages or phases?

Ashley: Yes, the training proceeded in three stages: warmup, dynamics pretraining, and depth injection.

During the warmup stage, they jointly optimized the pretrained backbone and the introduced prediction modules.

Then, in the dynamics pretraining stage, they trained the model to learn action-conditioned interaction dynamics.

Finally, in the depth injection stage, depth conditioning was introduced to refine the learned dynamics representations.

Evan: It sounds like a well-structured training process.

How did they ensure the integration of depth information into the model?

Ashley: Depth information was injected into the backbone through trainable fusion modules, which used token-wise routing for combining features across layers.

This cross-layer aggregated approach allowed the model to integrate depth information effectively without perturbs the existing features.

Evan: So what about the role of the queries in this model?

How are they conditioned?

Ashley: A set of N learnable world queries attends to hidden states from selected backbone layers.

These queries are conditioned through an action-conditioned adapter, which uses demonstrated controls to decode the future evolution of hand and scene dynamics.

Evan: And during deployment, how are these queries used by the policy?

Ashley: During deployment, the policy uses its own set of queries to read selected hidden layers, aligning with the frozen world queries to condition the action expert.

Essentially, these queries provide predictive supervision linked to real-time action generation.

Evan: That really helps in grasping the whole picture, Ashley.

Thanks for explaining the method section so thoroughly.

Ashley: You're welcome, Evan.

And that's the end of the Method section.

Let's move on to discuss the experiments and results to see how DexForesight performed.

Evan: Alright, now let's dive into the experiments and results section to see how DexForesight performed.

Ashley: The experiments are designed to answer three key research questions.

First, can DexForesight outperform existing baselines by retaining useful predictive structures without future decoding at deployment?

Second, is the predictive information learned by DexForesight’s world model accessible in the deployed policy representation?

And third, how do the components of the world model contribute to the policy's performance?

Evan: Let's start with the baseline comparison.

How did DexForesight perform against existing methods?

Ashley: In simulation, they evaluated DexForesight on 11 DexJoCo tasks, comparing it against two vision-language-action (VLA) policies, π0.5 and GR00T N1.7, and two world-action models, Fast-WAM and Cosmos3.

DexForesight reached an overall success rate of 60.5%, which is 9.6 percentage points above the strongest baseline, Fast-WAM, and 11.1 points above its own backbone, GR00T N1.7.

Evan: That's impressive!

Did it perform consistently across all tasks?

Ashley: Yes, it achieved the highest success rate in both single-arm and bimanual tasks, with 76.3% in single-arm tasks and 41.6% in bimanual tasks.

At deployment, DexForesight generates actions from policy queries and does not run the spatial world model or perform future decoding.

Evan: Wow, it sounds like DexForesight really shines when it comes to these simulations.

What about the real-robot evaluations?

Ashley: In real-robot evaluations, DexForesight was instantiated on the QwenVLA framework.

They selected three challenging tasks: Close Box, Insert Battery, and Screw On Lid.

Across these tasks, DexForesight achieved an overall success rate of 58.3%, which compared to 50.0% for WLA, 48.3% for GR00T N1.7, 30.0% for Cosmos3, and 10.0% for Fast-WAM.

Evan: It looks like DexForesight significantly outperforms these baselines.

How did they measure policy inference time?

Ashley: Under identical hardware and inference settings, the policy inference time for QwenVLA was 227.4 milliseconds, and it was 230.4 milliseconds for DexForesight, only a slight increase of 3 milliseconds, or 1.3%.

Cosmos3, on the other hand, took 930 milliseconds to generate a 16-frame future video during its video-generation stage.

Evan: So not only does DexForesight perform better, but it's also quite efficient.

What about the transfer of predictive information learned by the world model?

Ashley: They tested this with a spatial foresight probe, by freezing real-robot checkpoints and training a linear ridge probe to predict future 3D hand-point displacements at three different time horizons: 0.13, 0.27, and 0.54 seconds.

DexForesight achieved lower probing error compared to QwenVLA action-only model and GR00T N1.7, with the error decreasing from 13.59 to 10.01 millimeters at 0.54 seconds, a 26.3% reduction.

Evan: That’s quite a reduction.

Did they compare this approach with any direct methods?

Ashley: Yes, they compared DexForesight with direct additions to the GR00T policy, such as adding a DINO future-residual head, a hand-flow head, both heads together, and depth with Attention Residuals.

The hand-flow head was the strongest direct-prediction control at 53.1% overall.

However, DexForesight achieved 60.5%, surpassing this control by 7.4 points.

Evan: What does this suggest about their approach of using query alignment instead of direct supervision?

Ashley: It suggests that policy latents can retain predictive structure useful for control without future decoding, and that query alignment is more effective than directly supervising visual-change or hand-motion prediction.

Evan: What about other aspects of the world-model components?

How did features like offline depth and 3D hand-motion targets impact performance?

Ashley: They conducted several ablation studies to understand the impact of these components.

For example, without offline depth, the simulation success rate dropped by 10.8 points.

Replacing cross-layer fusion with context depth tokens yielded a success rate of 42.0%.

Replacing hand-point displacement with future-action targets dropped success to 48.4%.

These results support the importance of offline depth, cross-layer depth fusion, and kinematic hand-motion supervision for effective policy performance.

Evan: That’s quite a detailed analysis.

It seems like each component plays a crucial role in the performance of DexForesight.

Ashley: Evan.

These experiments and results demonstrate how DexForesight leverages implicit spatial foresight to significantly improve dexterous manipulation performance.

Evan: Thanks for explaining the experiments and results so clearly, Ashley.

Ashley: You're welcome, Evan.

And that's the end of the Experiments section.

Let's move on to the related work to understand the broader context of this research.

Evan: Alright, let's explore the related work section to understand how DexForesight fits within the broader landscape of robot policies and future modeling.

Ashley: The related work section highlights several key areas in the advancing field of dexterous manipulation and robot policies.

First, let's look at future modeling for robot policies.

Evan: Sure, Ashley.

How does future modeling typically interact with policy learning?

Ashley: World-action models generally couple future prediction with policy learning.

This can be done in two ways: by jointly modeling future observations and actions, or by incorporating predictive supervision during training without requiring future generation at deployment.

DexForesight belongs to the latter category, leveraging predictive supervision from its world model without decoding future outputs during deployment.

Evan: I see.

And what are some notable approaches in this area?

Ashley: One approach is EgoWAM, which explores 3D motion flow as a predictive target for human–robot co-training.

Other methods focus on image prediction and predictive features to structure robot policies.

However, these approaches often do not capture the coordinated motion of articulated fingers hidden from the cameras.

DexForesight addresses this by combining scene geometry with kinematic hand representations, jointly predicting hand motion and visual changes.

Evan: So, how does DexForesight's approach to spatially grounded policies stand out compared to others?

Ashley: DexForesight integrates geometric structure directly into the policy through its spatial world model.

This contrasts with methods that primarily focus on scene geometry, like ReconVLA and Spatial Forcing, which use auxiliary reconstruction to improve perception or align intermediate features with pretrained 3D foundation models.

DexForesight instead targets action-conditioned spatial dynamics, capturing hand–scene evolution beyond visible image geometry, including occluded finger motion.

Evan: That's a significant difference.

Prioritizing the articulated hand dynamics and scene changes seems crucial for dexterous manipulation.

Ashley: Exactly.

This predictive supervision, covering both scene and hand evolution, allows DexForesight to operate efficiently at deployment using only RGB, language, and robot state inputs.

Evan: How about the broader impact and contributions of world-action models in real-world robot applications?

Ashley: In addition to EgoWAM, approaches like Fast-WAM and Cosmos3 have significantly contributed to the field by enabling predictive features and future generation for policy learning.

Fast-WAM specifically focuses on eliminating the need for test-time future imagination, whereas Cosmos3 generates future images and actions jointly.

DexForesight extends these ideas by focusing on implicit spatial foresight, simplifying deployment without sacrificing predictive accuracy.

Evan: It's interesting how DexForesight builds on and refines these existing approaches.

What about the role of auxiliary tasks like reconstruction in enhancing spatial perception?

Ashley: Auxiliary tasks indeed play a significant role in enhancing spatial perception.

For instance, ReconVLA uses auxiliary reconstruction to improve spatial perception, and Spatial Forcing aligns intermediate features with pretrained 3D models to enable implicit spatial grounding.

DexForesight leverages predictive supervision from its spatial world model to achieve similar benefits without needing explicit reconstruction tasks at deployment.

Evan: That's a smart way to utilize spatial information.

By aligning predictive features with action learning, DexForesight seems to streamline the whole process efficiently.

Ashley: Yes, integrating predictive spatial dynamics directly into policy latents represents a remarkable advancement.

This helps address challenges related to occluded finger motion and scene changes, crucial for dexterous manipulation.

Evan: Well, that gives us a solid understanding of the related work and how DexForesight innovates within this space.

Thanks for breaking down these comparisons, Ashley.

Ashley: You're welcome, Evan.

And that wraps up the Related Work section.

Let's move on to discuss the conclusion and broader implications of DexForesight's contributions.

Evan: Alright, Ashley, we've covered the background, methodology, experiments, and related work for DexForesight.

Let's summarize the key contributions and takeaways from this paper.

Ashley: Evan.

DexForesight introduces a novel framework for learning spatial foresight implicitly.

By grounding dexterous policies with predictive representations from a frozen action-conditioned spatial world model, it significantly enhances the efficiency and performance of robotic hand manipulation.

Evan: And its success rate improvements are noteworthy.

Can you recap the main experimental findings?

Ashley: Of course.

In simulation, DexForesight improved the overall success rate from 49.4% to 60.5%, outperforming strong baselines like Fast-WAM and Cosmos3.

For real-robot tasks, it achieved an impressive 58.3% success rate compared to 50.0% for the strongest baseline.

These results underscore the importance of implicitly learned spatial foresight in handling complex dexterous manipulation tasks.

Evan: It's clear that DexForesight offers substantial advancements.

What about the methodology's efficiency during deployment?

Ashley: DexForesight operates efficiently at deployment using only RGB observations, language, and robot state without the need for depth input or explicit future prediction.

This streamlined approach is key to its practical utility in real-world applications.

Evan: That's definitely an important aspect.

The ability to retain complex predictive knowledge without computational overhead during deployment is a big win.

Ashley: Indeed, Evan.

By integrating predictive hand–scene dynamics directly into policy latents, DexForesight represents a significant step forward in the field of dexterous manipulation.

It's a testament to how spatial foresight can be leveraged without adding complexity to deployment.

Evan: Listeners, that brings us to the end of today's episode of Daily Paper Cast.

Thank you for joining us as we explored the innovative DexForesight framework.

Ashley: We hope you found the discussion insightful and are inspired by the advancements in dexterous robotic manipulation.

Evan: Be sure to tune in next time as we bring you more cutting-edge research from the world of AI, robotics, and machine learning.

Ashley: Thank you for listening, and we look forward to having you back for our next episode.