JEPA-Anything: Learning Predictive Models across Different Worlds

Evan: Welcome to Daily Paper Cast.

Evan: Today's paper is from the Hugging Face daily paper list of September 18, 2026, with 21 upvotes.

Ashley: The title is JEPA-Anything: Learning Predictive Models across Different Worlds.

Evan: The first two authors are Taoyong Cui and Zhongyao Wang, and the corresponding author is Yingcheng Wu from PhAI Labs.

Ashley: Let's dive into the Introduction section, Evan.

This paper is all about world modeling, which enables intelligent systems to anticipate the consequences of environments they interact with.

Evan: Exactly, Ashley.

Early latent world models were mainly designed to compress observations and understand the dynamics of system behaviors for control tasks.

But the researchers are asking a broader question here, aren't they?

Ashley: Yes, they are exploring if a common learning principle can support world modeling across radically different systems.

Traditionally, predictive models have been domain-specific, but this work extends the concept to be domain-agnostic using a framework called JEPA-Anything.

Evan: Interesting.

So what's novel about JEPA-Anything?

Ashley: JEPA-Anything is based on something called orthogonal predictive factorization, or OPF.

Extending joint-embedding predictive architectures, OPF decomposes latent targets into complementary factors, learns them through dedicated pathways, and then recombines them to form a shared predictive design.

This allows the framework to adapt to diverse systems.

Evan: That's quite a leap.

What kinds of domains did they test JEPA-Anything on?

Ashley: They evaluated it across seven domains: vision, biology, clinical trajectories, control tasks, molecular dynamics, physical fields, and weather.

They were practically asking if the same predictive model could adapt and perform well across these very different areas.

Evan: Did they specify any unique tasks associated with these domains?

Ashley: Experiments included representation learning, intervention prediction, out-of-distribution generalization, and long-horizon dynamics.

Some examples are forecasting over 1,000 clinical events and performing 100-step molecular rollouts across four systems.

Evan: And how did JEPA-Anything perform against benchmarks?

Ashley: It improved the reported metrics on all 10 dynamics tasks matched against JEPA baselines and reduced the single-intervention prediction error on Interventional Pong by 34.8 percent.

It also achieved the lowest one-step and 100-step molecular prediction errors among compared methods in all four systems.

Evan: That’s impressive.

What about beyond pure prediction?

Did they delve into any real-world applications?

Ashley: Yes, they did.

For example, a factor-nominated biological intervention gained experimental support in cell co-cultures, patient-derived organoids, tumor fragments, and mice.

Moreover, they demonstrated that their system could recover a physical law, specifically the Keplerian scaling exponent, with a fitted slope of negative 1.4991.

Evan: So, essentially, JEPA-Anything not only shows promise in predictive performance but also connects predictive modeling with real-world intervention and scientific discovery.

Ashley: Exactly.

Their results support the idea that a common factorized predictive principle can be effective across heterogeneous worlds.

Evan: That's the end of the Introduction section.

Let’s move on to the detailed methods next.

Evan: Alright Ashley, let's dive into the Method section of this fascinating paper.

Ashley: Sure, Evan.

The core of JEPA-Anything is its orthogonal predictive factorization, or OPF.

The researchers propose a framework that separates domain-specific components from a shared predictive core.

Evan: So, how exactly does this OPF mechanism work?

Ashley: Great question.

OPF involves decomposing a latent target into multiple complementary factors.

Each of these factors is learned through dedicated pathways, and then they’re recombined to form a complete latent state prediction.

Evan: Does this mean they are assigning specific parts of the prediction task to different parts of the model?

Ashley: Precisely.

By doing this, they optimize the allocation of predictive capacity.

For example, one factor might capture local details while another grasps global patterns.

This mitigates the risk of overfitting to high-variance structures or trivial components.

Evan: Interesting.

How do they ensure that these factors don't overlap or become redundant?

Ashley: They introduce orthogonality constraints within and across these factors.

Essentially, this means that the predictive pathways are trained not to learn overlapping or redundant information.

They use within-factor and cross-factor orthogonality objectives to achieve this.

Evan: Seems like a smart way to maintain diversity in predictive factors.

But how do they synthesize these factors back into a complete state?

Ashley: The synthesis uses the Moore-Penrose pseudoinverse of the analysis map, where the predicted components are recombined into a full latent state.

This process ensures that the synthesized latent state remains accurate and useful for downstream tasks.

Evan: And what's the advantage of using this pseudoinverse synthesis?

Ashley: The main advantage is that it preserves both direction and magnitude required for accurate state synthesis.

This step is crucial for ensuring the predictive state can handle multiple downstream tasks effectively.

Evan: Okay, but what about the practical implementation?

How do they tune this system for different domains?

Ashley: Good point.

The adaptability of JEPA-Anything comes from domain adapters and view samplers.

An adapter maps raw observations into a shared token and structural descriptor format, which the predictive core can then process.

View samplers handle context-target selection, which also varies by domain.

Evan: Could you give an example of such an adapter?

Ashley: Sure.

For instance, in a visual domain, an adapter might tokenize image patches and encode positional descriptors, while in a clinical domain, it might include patient history and demographic data.

Evan: Got it.

Once these tokens and descriptors are set up, what comes next in their pipeline?

Ashley: Once they have the context and target tokens, the online encoder generates a context representation, whereas the EMA target encoder generates latent targets.

These targets are then predicted through dedicated pathways, and the whole setup is trained with domain-specific base losses and the additive OPF regularization terms.

Evan: Additive OPF regularization terms?

What are those?

Ashley: These are regularization terms specific to orthogonality, factor activity, and encoder variance.

They ensure that different predictive factors remain active and diverse, and help avoid any collapse in representation.

Evan: And the final synthesized state can then be used for what exactly?

Ashley: The synthesized state can support various tasks.

For terminal readout scenarios, it provides a stable representation for downstream tasks.

In recursive tasks like intervention-conditioned predictions or planning, it feeds into further transitions.

For factor-level scientific analysis, it can be used to diagnose specific predictive pathways.

Evan: It sounds like JEPA-Anything covers a lot of ground.

How do they handle factor activity and ensure everything remains efficient?

Ashley: They enforce an activity floor for each factor and use a variance term to keep the online encoder's representations from collapsing.

This means each predictive factor remains active and contributes uniquely to the model's predictions.

Evan: This sounds great in theory.

But how effective is it in practice?

How do they evaluate JEPA-Anything?

Ashley: They evaluate it across three main groups: terminal readout, latent world dynamics, and scientific analysis.

Terminal readout tests visualize binding and single-cell tasks; latent world dynamics cover intervention-conditioned sequences, control tasks, and long-rollout stability; while scientific analysis includes factor-level diagnostics and experimental validation.

Evan: So, they basically test its performance in quite varied settings to show its adaptability and robustness?

Ashley: Indeed.

This multi-faceted evaluation aims to demonstrate that JEPA-Anything can truly generalize across domains while maintaining predictive accuracy.

Evan: Gotcha.

And that's the end of the Method section.

Next up, we'll dive into their experimental results.

Evan: Now, let's discuss the Experiments and Results section of the paper, Ashley.

Ashley: Evan.

The authors evaluated JEPA-Anything across three broad groups: terminal readout, latent world dynamics, and scientific analysis.

These groups encompass a variety of domains and tasks.

Evan: That’s a comprehensive approach.

What did the first group, terminal readout, focus on?

Ashley: The first group focused on evaluating the predictive state at a terminal downstream readout.

They used three primary experiments: controlled visual binding on MuJoCo scenes, single-cell state representation, and longitudinal disease-state forecasting.

Evan: What can you tell us about the controlled visual binding experiment?

Ashley: In this experiment, they tested whether OPF improved the compositional readout of controlled visual changes.

They used a visual dataset from MuJoCo scenes and evaluated JEPA-Anything against standard JEPA and some frozen checkpoints.

Evan: And, how did JEPA-Anything perform?

Ashley: JEPA-Anything increased the injective held-out-cell accuracy and grid recovery while reducing the collapse rate compared to both DINOv3 and SigLIP2 baselines.

Evan: That’s impressive.

What about the single-cell state representation?

Ashley: They used single-cell RNA sequencing data and evaluated the models on PBMC cell-type clustering and perturbation-response prediction.

JEPA-Anything outperformed previous models on both clustering metrics and perturbation-response datasets.

Evan: And in the case of longitudinal disease-state forecasting?

Ashley: For this, they used a longitudinal multimodal cohort to predict over 1,000 future clinical events.

JEPA-Anything achieved a higher mean precision-recall area under the curve (PRAUC) than matched JEPA.

Evan: How fascinating.

Now, what about the second group, latent world dynamics?

Ashley: This group evaluated the world-modeling capability of JEPA-Anything, including intervention-conditioned compositional prediction and out-of-distribution and long-horizon dynamics.

They used datasets from CITRIS Interventional Pong, CausalWorld, DeepMind Control Suite, PDEBench, and WeatherBench2.

Evan: That sounds extensive.

Can you explain the intervention-conditioned prediction results?

Ashley: Sure.

JEPA-Anything significantly reduced the mean squared error (MSE) for single-intervention and combined-intervention one-step predictions compared to dense standard JEPA.

It also performed better in six-step free rollouts, showing superior compositional reuse of learned changes.

Evan: What about their findings on out-of-distribution and long-horizon dynamics?

Ashley: JEPA-Anything performed more accurately on unseen initial conditions and held-out episodes, indicating that it learned reusable transition rules.

Additionally, it exhibited lower error growth across multiple rollout steps in varied tasks including robot states, pixel dynamics, and continuous physical fields.

Evan: And how did it fare in continuous-control tasks?

Ashley: The researchers evaluated it using model-based control tasks like Hopper, Walker2d, and HalfCheetah.

JEPA-Anything achieved higher cumulative rewards in Walker2d and HalfCheetah, although Hopper slightly favored standard JEPA.

Evan: It seems JEPA-Anything demonstrated strong capabilities in diverse settings.

What about the third group, scientific analysis?

Ashley: The third group focused on using factor coordinates for scientific analysis.

They evaluated factor-nominated interventions in a biological setting and examined predictive latent modes in an orbital scaling law scenario.

Evan: Can you elaborate on the biological intervention findings?

Ashley: Certainly.

JEPA-Anything was used to nominate an intervention combining IL-18 with NT5E/CD73 blockade.

Wet-lab evaluations showed that this combination enhanced antitumor activity in various preclinical models, including co-cultures, organoids, tumor fragments, and immunocompetent mice.

Evan: What about their analysis of orbital scaling laws?

Ashley: In the orbital case, JEPA-Anything’s learned spectral modes recovered the Keplerian scaling exponent with high precision, demonstrating the model's ability to capture physically meaningful dynamics.

Evan: That’s quite a testament to the model's versatility and accuracy.

Any final highlights from the results?

Ashley: Overall, JEPA-Anything consistently outperformed standard JEPA across a variety of tasks, showing robustness, generalizability, and the additional benefit of interpretable and relevant factor-level diagnostics.

Evan: That's the end of the Experiment section.

Next, we'll discuss the related work.

Evan: Alright, Ashley, now let's delve into the Related Work section of the paper.

Ashley: Sounds good, Evan.

The first area the authors discuss is latent world models and planning.

These models compress observations into states that can be advanced using learned dynamics.

Evan: Ah, I see.

We've seen early models like recurrent world models that couple latent dynamics with controllers.

Any recent developments?

Ashley: Yes, for instance, DreamerV3 has demonstrated that a single model-based reinforcement learning configuration can support behavior learning across various domains.

Evan: And how about approaches that do not rely on decoders?

Ashley: Decoder-free approaches like TD-MPC2 directly optimize latent dynamics for scalable continuous control.

Another interesting work is DINO-WM, which predicts features from a pretrained visual encoder and plans towards visual goals from offline trajectories.

Evan: Interesting.

And I believe V-JEPA 2 combines large-scale representation learning with action-conditioned latent models for robotic planning?

Ashley: Exactly.

These systems primarily focus on developing latent states for visual prediction or control.

Evan: How does joint-embedding predictive learning fit into the picture?

Ashley: Joint-embedding predictive learning, or JEPL, involves predicting representations of masked image regions or video frames from visible context.

For example, I-JEPA predicts representations of masked image regions, while V-JEPA extends this to video without pixel reconstruction.

Evan: V-JEPA, interesting.

Have there been any extensions to physical understanding and planning?

Ashley: Yes, recent work scales JEPL principles to physical understanding and planning.

For instance, Cell-JEPA adapts latent prediction to single-cell transcriptomics.

Evan: But what kinds of learned invariances depend on these JEPL objectives?

Ashley: Analyses show that the learned invariances from JEPL objectives depend on which signals vary across the context-target pairs.

This is crucial for understanding the dependencies and structure within the data.

Evan: That makes sense.

What about structured and factorized representations?

How do they relate to JEPA-Anything?

Ashley: Structured and factorized representation learning often seeks coordinates aligned with independent generative variables, like in beta-VAE, or entity-aligned slots, like in Slot Attention.

Similarly, structured world models represent states as interacting objects and relations, while CITRIS uses temporal interventions for causal factor identification.

Evan: Those approaches sound quite specific.

Does JEPA-Anything require similar inductive biases or supervision?

Ashley: Not necessarily.

JEPA-Anything uses orthogonal predictive factorization to decompose target capacity without assuming object, causal, or semantic factor identities.

It applies orthogonality and activity constraints to the learned target-subspace bases, and retains an explicit synthesis map.

Evan: Aha, I see.

Then how about redundancy reduction and collapse prevention?

Any connections there?

Ashley: Definitely.

Redundancy reduction and collapse prevention methods like Barlow Twins reduce redundancy through cross-correlation matching.

VICReg combines invariance with explicit variance and covariance regularization.

Evan: And how does OPF fit within this context?

Ashley: OPF factorizes predictive target capacity without predefined semantics and applies orthogonality and activity constraints.

This links redundancy reduction to creating a complete latent world state, ensuring diverse and informative representations.

Evan: Very comprehensive, Ashley.

JEPA-Anything seems to draw from a broad range of related works to build a robust and flexible predictive model.

Ashley: Indeed, it does.

And that's the end of the Related Work section.

Evan: Alright Ashley, let's wrap up this episode with a summary of the key contributions and takeaways from this paper.

Ashley: JEPA-Anything makes significant strides in creating a domain-agnostic predictive modeling framework.

One of its key innovations is the orthogonal predictive factorization, or OPF, which decomposes latent targets into complementary factors, each learned through dedicated pathways.

Evan: Right.

And these orthogonal factors help in improving predictive accuracy while ensuring diversity in learned representations.

This is crucial for scaling predictive modeling across different domains.

Ashley: Precisely.

JEPA-Anything was evaluated across seven diverse domains, including vision, biology, clinical trajectories, control tasks, molecular dynamics, physical fields, and weather.

It consistently outperformed standard JEPA in terms of predictive metrics and demonstrated robust generalization capabilities.

Evan: The paper provided impressive results, such as reducing single-intervention prediction error by 34.8 percent and achieving the lowest errors in one-step and 100-step molecular predictions among compared methods.

Ashley: Beyond predictive performance, JEPA-Anything effectively connected predictive modeling with real-world intervention and scientific discovery, like validating biological interventions in wet-lab experiments and recovering physical laws from latent orbital modes.

Evan: So, the key takeaway here is that JEPA-Anything offers a flexible, scalable, and accurate predictive framework that can be applied across a wide range of domains, paving the way for new interdisciplinary advancements.

Ashley: Indeed.

It's an exciting step forward for predictive modeling, demonstrating that a common factorized principle can bridge diverse scientific and practical applications.

Evan: Thank you for tuning into Daily Paper Cast.

We hope you found this episode informative and engaging.

Ashley: Don't forget to join us again for another deep dive into cutting-edge research.

Until next time, stay curious and keep exploring.