JEPA: From Language Models to World Models

A language model can tell you what usually comes next. A world model should tell you what happens if you act. That is the core of Yann LeCun’s JEPA bet.

Asked whether JEPA or world-model approaches will replace LLMs, LeCun answered: “Eventually they replace LLMs,” because LLMs are “really good at manipulating language but basically nothing else” [9]. In a second conversation, asked whether JEPA-based approaches will overtake vision-language-action models, he went further: “Oh, absolutely. Yeah, VLA are doomed” [10].

I would not go that far yet. LLMs already do useful work far beyond chat, and today’s VLA systems are much more impressive than the word “doomed” suggests. The useful part of LeCun’s provocation is architectural: should the core of an intelligent system be a language model that predicts tokens, or a world model that predicts state? That is why JEPA is interesting. The critique is not that language is useless. Language is how humans instruct, inspect, and coordinate with machines. The critique is that next-token prediction may be a strange training signal for agents that need to perceive, predict, act, and stay safe in changing environments.

In simplified form, the language-model loss says: given the previous tokens, assign high probability to the actual next token.

$$\mathcal{L}_{LM} = -\log p(x_t \mid x_{<t})$$

The equation above penalizes the model when the true next token $x_t$ is unlikely under the context $x_{<t}$. This simple rule has produced surprisingly general behavior. It taught models grammar, facts, style, some reasoning patterns, tool use, and a lot of compressed world knowledge. It is one of the great empirical surprises in machine learning. But an agent needs to answer a different set of questions: What state am I in? What is likely to happen next? What changes if I act? Which future state should I aim for? What subgoal should I reach before the final goal? Those are not primarily language questions. They are state questions. JEPA points at a different training target: predict the latent state that makes future perception and action useful. If that works, future systems may organize around world models rather than language models.

Why next-token prediction got us this far

Before getting into JEPA, it is worth being fair to LLMs. Next-token prediction is brutally scalable: every document, code file, transcript, and forum thread becomes labeled training data. It also works because language is already compressed human experience. Text contains physics, social behavior, software conventions, recipes, plans, contracts, emotions, and arguments.

That is why LLMs became useful general interfaces. The mismatch appears when we ask them to become agents. Token prediction can still produce useful hidden states, but the training signal only checks whether the next visible token was likely. It does not directly check whether the model represented the physical state, the future consequences, or the variables needed for control. For an agent that has to perceive, predict, act, and correct itself over time, that gap can become the bottleneck.

Token prediction can be locally right and globally weak

A language model can generate the right next token while still having a weak internal trajectory for what comes next. That sounds subtle, but it matters. A token is a local target. A world state is a stronger constraint. If the model says the right word while its hidden state drifts into a bad part of representation space, the immediate output may look fine while the system is poorly positioned for planning, consistency, or future action.

The Semantic Tube Prediction paper is useful here because it separates two things that are easy to collapse: producing the right next token and maintaining a useful hidden-state trajectory. The paper argues that a model can land in the correct token region while drifting away from the representation path that would support later predictions [1]. One of its core assumptions is:

“The trajectory of $x_{\leq t}$ is locally linear almost everywhere.” [1]

The narrow point is that the next-token loss can reward the right next word without guaranteeing that the hidden trajectory remains useful for later prediction.

This is the first intuition behind JEPA: the target should not only be the next visible thing. The target should shape the internal state that makes future prediction useful. If the goal is a better autocomplete system, predicting tokens is natural. If the goal is an autonomous system that can handle changing environments, future consequences, and multi-step plans, the hidden state itself probably needs to become a first-class training target.

JEPA changes the object of prediction

A JEPA-style model predicts the representation of another view, missing part, or future state. It does not need to reproduce every pixel or emit every word. It learns an encoder and a predictor so that one latent representation can predict another. A simple version looks like this:

$$z_c = E(x_{context})$$ $$z_t = E_{target}(x_{target})$$ $$\mathcal{L}_{JEPA} = D(P(z_c), \text{stopgrad}(z_t)) + R(E)$$

Here, $E$ encodes the context, $E_{target}$ encodes the target view or target state, $P$ predicts the target representation, $D$ measures distance in latent space, and $R(E)$ keeps the representation from collapsing. That anti-collapse term is not a detail. It is the difference between learning a useful latent space and mapping everything to the same vector.

A naming note before going further: I use JEPA for the broad joint-embedding predictive idea. LeJEPA is a specific JEPA variant from a separate paper [3]. Its main difference is the anti-collapse mechanism: it adds SIGReg, a regularizer that pushes embeddings toward an isotropic Gaussian. I use LeJEPA when discussing that specific regularization and the theory built around it, not as a synonym for every JEPA model.

This is the core shift:

The important part is not that JEPA uses embeddings. Everyone uses embeddings. The important part is that JEPA trains the embedding space to be predictive. V-JEPA 2.1 defines JEPA as a framework that learns representations “by making predictions in a learned latent space, rather than directly in the observation (input) space” [4]. That sentence is the architectural pivot.

For LLMs and most VLMs, the training check is still token-level: did the model assign enough probability to the next answer token? JEPA changes the check. Given a context view, can the model predict the target embedding for another view, masked region, or future state? That does not solve agency by itself, but it puts pressure on the representation to carry information that survives across views and time. For a robot or long-running visual agent, that can include object position, pose, contact, reachable surfaces, and whether the scene is moving closer to the goal.

What changes for vision-language models

This is where the JEPA argument becomes more concrete. Most VLMs turn perception into text. The model sees an image or video, takes a question or instruction, and produces tokens. That is useful, but it puts language generation in the loop even when the system mainly needs an updated state.

A vision-language JEPA changes the default object being predicted. Instead of mapping every visual question into an autoregressive text sequence, it can predict the semantic target embedding directly. VL-JEPA makes this argument clearly. The paper says:

“Instead of autoregressively generating tokens as in classical VLMs, VL-JEPA predicts continuous embeddings of the target texts.” [8]

That matters because the world is underdetermined at the surface. Ask a model what happens if a switch is flipped down. “The lamp turns off,” “the room goes dark,” and “the light shuts off” can all be correct. In token space, those answers are different strings. In semantic embedding space, they should be nearby states. VL-JEPA uses the same example: “the lamp is turned off” and “room will go dark” are separate in raw token space, but ideally close in embedding space [8].

The implication is bigger than parameter efficiency. A VLM can become a continuous perception system. It can watch, update a semantic state, compare that state to a query or goal, and decode text only when language is actually needed. VL-JEPA reports roughly 50% fewer trainable parameters in a controlled comparison and about 2.85x fewer decoding operations with selective decoding while maintaining similar output quality [8]. The exact numbers will change with models and tasks, but the direction is important: language becomes an interface to a visual state, not the only form the state can take.

That changes what VLMs are good for. Instead of treating every frame as more context for a token generator, a JEPA-style VLM can maintain a compact, queryable representation of what is happening. For video systems, that is a major shift. A model watching a long stream should not need to narrate everything. It should keep track of the meaningful state changes and speak when the task requires it.

A world model is more than an embedding

Calling JEPA a world-model approach raises the bar. In his 2022 position paper, LeCun framed common sense as “a collection of models of the world” that tell an agent what is likely, plausible, and impossible [11]. That standard shifts attention from whether a model has embeddings to what those embeddings preserve. A representation can be useful for retrieval or classification and still drop the state variables needed for control. For agents, the latent space has to preserve the variables that matter for prediction and action.

A useful world model needs at least four things: a compact state representation, a way to predict future states, a way to condition those predictions on actions, and a planner that can search over possible futures. The strongest theoretical idea here is linear identifiability. If $z$ is the true latent state of the world, a good learned representation $h(z)$ should recover it up to a simple transformation:

$$h(z) = Qz$$

Here $Q$ can be a rotation or another simple linear transform. The exact coordinates may change, but the geometry should remain usable. This matters because planning is geometry. If the real world has a smooth path from state A to state B, but the learned latent space twists that path into something warped, a planner operating in latent space will make confident mistakes. It may optimize the wrong distance or choose a straight line in embedding space that becomes a bad trajectory in the real world.

A theory paper on LeJEPA makes this precise: under Gaussian latent variables, OU-style transitions, alignment, and Gaussian regularization, the learned representation can recover the true latent state up to rotation [2]. That is a narrow theorem about a specific setup, not a universal claim about all JEPAs. But it gives the right test: a world model is not an embedding that “looks semantic.” It is an embedding whose geometry preserves the world well enough for prediction and planning.

This is the cleanest way to state the embedding-space objection: JEPA’s biggest advantage is also its biggest risk, because everything important happens in latent space. If the latent space is faithful, JEPA gives agents a compact substrate for prediction. If it is collapsed, over-compressed, or geometrically wrong, the system can fail while looking mathematically elegant.

The anti-collapse problem is the whole game

A naive latent-prediction model has an easy way to win: make every embedding the same. If $E(x) = c$ for every input, then predicting the target embedding is trivial. The loss can look good while the representation is useless.

This is why JEPA methods care so much about stop-gradients, target encoders, exponential moving averages, whitening, variance constraints, contrastive losses, or explicit distributional regularizers. They are what prevent the model from cheating.

The LeJEPA paper’s answer is SIGReg: push the distribution of embeddings toward an isotropic Gaussian [3]. The intuition is that a good latent space should be spread out, statistically well-conditioned, and difficult to collapse. The paper emphasizes that LeJEPA combines prediction with a regularizer that makes the embedding distribution behave well, reducing reliance on a bag of heuristics [3].

This also explains why “everything happens in embedding space” should not be dismissed as hand-waving. Embedding space is not magic. It has to be engineered, constrained, and tested against whether it preserves the right variables. JEPA is powerful only if the learned space is shaped by objectives that make prediction, planning, and control possible. A bad latent space is worse than a bad image. At least a bad image can be inspected; a bad latent space may fail silently.

Video makes the argument obvious

Language hides this problem because text is already an abstraction. Video makes it obvious. A video is not just a list of frames. It is a stream of state changes: objects move, hands interact with objects, cameras shift, and actions create consequences.

A pixel generator can learn to produce plausible future frames. That is useful, but it forces the model to spend capacity on surface detail. LeCun’s dashcam example is perfect: a generative video model may waste resources predicting the random motion of leaves beside the road because those leaves occupy many pixels, even though they are mostly irrelevant to driving [9]. A JEPA-style video model asks a different question: how should the representation move?

$$\hat{z}_{t+1} = z_t + \Delta z_t$$

That is the right kind of abstraction for agents. The model does not need to render every texture to understand that an object moved left, a hand approached a cup, or a door is now more open. V-JEPA 2.1 pushes in this direction by predicting masked or future visual representations, making features spatially dense, and preserving temporal consistency [4]. The paper’s target is not just semantic recognition. It tries to make the latent state “spatially structured, semantically coherent, and temporally consistent” [4].

That phrase matters. A robot does not only need to classify a scene. It needs to know where things are, how they move, which surfaces matter, and which changes persist across time. If a model sees a hand move toward a cup, the most important prediction is not the exact next pixel color of the hand. It is the evolving state: distance to cup, grasp possibility, object pose, likely contact, future occlusion, and maybe the intention implied by the motion. Those are latent variables, and this is where JEPA starts to look like a bridge from perception to action.

What changes for vision-language-action models

VLMs connect perception to language. VLAs connect perception, language, and action. A VLA can read an instruction, look at a workspace, and output motor commands or action tokens. That is already a large step beyond captioning. The question is whether the action system has a reusable model of consequences, or whether it is mostly learning a direct mapping from observation and instruction to action.

JEPA points to a different middle layer for VLAs. The stack becomes: perceive the scene, encode the current latent state, predict how candidate actions change that state, then choose actions that move the world toward a goal. Language still matters because it specifies goals, constraints, and explanations. But the control loop needs a state space where consequences can be predicted before the robot acts.

This is the difference between a VLA that reacts and a VLA that can plan. If the instruction is “put the cup in the drawer,” a direct VLA policy may learn useful behaviors from demonstrations. A JEPA-style world model should also represent intermediate states: the cup is visible, the gripper is aligned, the cup is graspable, the drawer is open, the cup is above the drawer, the cup is released. Those are not just words. They are latent states the system should recognize, predict, and reach.

The implication for multimodal AI is practical. Bigger context windows help a VLM remember more frames. They do not by themselves give the model action-conditioned dynamics. More demonstrations help a VLA imitate more behaviors. They do not by themselves give the system a compact space for counterfactual search. JEPA is interesting because it tries to make the hidden state itself predictive enough to support planning.

Action is the line between representation and agency

A model that predicts what happens next from observation alone is still missing a key question for agency: what changes if the system chooses an action? For a world model, the core equation is:

$$z_t = E(o_t)$$ $$\hat{z}_{t+1} = F(z_t, a_t)$$

The model observes $o_t$, encodes it into latent state $z_t$, then predicts the next latent state after action $a_t$. That changes the question from “what comes next?” to “what happens if I do this?” This is the line LeCun keeps drawing around real agency: “I do not understand how you can even think of building an agentic system without the ability to predict the consequences of its actions” [10].

LeWorldModel is a concrete example of this direction. It learns from pixels, predicts next latent states conditioned on actions, and plans by rolling out candidate actions in latent space [5]. The paper states the JEPA instinct clearly:

“Instead of attempting to model every aspect of the environment, JEPA focuses on capturing the most relevant features needed to predict future states.” [5]

The planner can optimize toward a goal embedding:

$$a^*_{1:H} = \arg\min_{a_{1:H}} D(\hat{z}_{t+H}, z_{goal})$$

This is still early. It is bounded, short-horizon, and evaluated in controlled settings. But architecturally it is the right shape. The model does not need to generate a full video of the future. It needs to predict the state variables that make action selection possible.

This is why JEPA might replace language models as the core of some future AI systems. Language is still useful for instructions, tools, and explanation. But an agent that lives in the world needs a predictive state engine.

Long-horizon autonomy is a decomposition problem

If a task requires many steps, a flat planner has two problems. Prediction error compounds with every rollout step, and the search space grows with the horizon. The Hierarchical World Models (HWM) paper [6] says single-level planning fails in two regimes: non-greedy tasks and long-horizon tasks where “prediction errors compound over autoregressive rollouts and the action search space grows exponentially with horizon.”

Hierarchical latent world models attack this directly. The high-level model plans over coarse future states or macro-actions. Its first predicted future state becomes a subgoal. The low-level model then plans primitive actions to reach that subgoal. The system replans repeatedly as new observations arrive.

$$z_{subgoal} = F_{high}(z_t, l_t)$$ $$a^*_{1:h} = \arg\min_{a_{1:h}} D(F_{low}(z_t, a_{1:h}), z_{subgoal})$$

The high-level model handles long-horizon direction while the low-level model handles short-horizon precision. LeCun’s version is simple: low levels make “short-term prediction with a lot of details,” while longer-term prediction has to throw away detail so it does not diverge from reality [10].

This is one of the strongest arguments for JEPA-style world models. In the HWM paper, hierarchy improves planning across latent world-model backbones and task suites. One reported robot result is especially sharp: under the evaluated setup, flat VJEPA2-AC gets 0% success on Franka pick-and-place, while the hierarchical version reaches 70% for the cup task [6].

The failure is not only representation quality. Manual subgoals can rescue flat planners, which means the bottleneck is often decomposition.

This is also where language-only planning starts to feel brittle. A chain-of-thought can describe substeps, and a task list can store a plan. But an embodied agent needs subgoals in the same space where it predicts consequences. LeCun puts the point bluntly: “your cat can do hierarchical planning, and your cat does not know language” [10]. The point is not that cats are the benchmark, but that hierarchical planning does not have to be expressed first as language. A useful agent should be able to represent “the cup is in a graspable pose” not merely as a sentence, but as a latent state it can recognize, predict, and reach.

The risk: elegant latent spaces can lie

This is the part that should make us cautious. JEPA only works if the latent space preserves the right structure. There are many ways to fail:

• Collapse: everything maps to the same point.
• Over-compression: useful details disappear.
• Wrong variables: the representation captures texture instead of state.
• Warped geometry: latent distances do not match controllable changes.
• Off-manifold planning: the planner searches states the model never learned.
• Weak action coverage: the model cannot predict actions outside its data.
• Missing hierarchy: short-horizon predictions do not compose into long-horizon behavior.

This is also where human-like intelligence comparisons should be handled carefully. Humans do not merely compress the world into minimal vectors. We preserve messy, useful detail. We keep context, remember exceptions, and carry “inefficient” structure because it helps us adapt. The From Tokens to Thoughts paper makes a related point about LLM embeddings: they “broadly align with human category boundaries, yet fall short on fine-grained semantic distinctions” [7]. A JEPA-style system that compresses too aggressively may be efficient and still miss what matters.

The core question becomes: which constraints make embeddings preserve the world variables needed for action? That is where the field is heading: better objectives, better anti-collapse methods, better temporal structure, action-conditioned prediction, hierarchy, memory, and meta-control.

The likely future stack

I do not think the future is a giant JEPA that simply replaces every language model use case. A more plausible architecture is simpler: language remains the interaction layer, while a JEPA-style world model holds and predicts state.

At a high level:

• Encoders map observations and instructions into latent state.
• A JEPA-style world model predicts how that state changes.
• A planner searches over future states and selects a path.
• Decoders turn the result into human-readable language or action-readable commands.

The important point is where language sits in the system. Language remains how humans instruct, inspect, and coordinate with machines. It is also how the system explains itself back to us. If the latent space becomes faithful enough, future agents may use language as the interface over a predictive world model: encode observations into state, predict how the state changes, plan over that state, and decode the result into words or actions.

References

[1] Semantic Tube Prediction: Beating LLM Data Efficiency with JEPA — https://arxiv.org/pdf/2602.22617
Used for the “correct token, wrong hidden-state trajectory” framing and the opening illustration.

[2] When Does LeJEPA Learn a World Model? — https://arxiv.org/pdf/2605.26379
Used for linear identifiability, latent geometry, world-model criteria, and embedding-space caveats.

[3] LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics — https://arxiv.org/pdf/2511.08544
Used for the LeJEPA variant, SIGReg, anti-collapse framing, and embedding-space caveats.

[4] V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning — https://arxiv.org/pdf/2603.14482
Used for latent-space video prediction, dense/spatiotemporal representation claims, and the JEPA mechanism illustration.

[5] LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels — https://arxiv.org/pdf/2603.19312
Used for action-conditioned latent dynamics, planning from pixels, and the planning illustration.

[6] Hierarchical Planning with Latent World Models — https://arxiv.org/pdf/2604.03208
Used for hierarchy, macro-actions, subgoals, long-horizon planning, and the hierarchy illustration.

[7] From Tokens to Thoughts: How LLMs and Humans Trade Compression for Meaning — https://arxiv.org/pdf/2505.17117
Used for the compression vs semantic nuance caveat.

[8] VL-JEPA: Joint Embedding Predictive Architecture for Vision-language — https://arxiv.org/pdf/2512.10942
Used for the vision-language JEPA contrast with token-generative VLMs, the light-switch example, and selective decoding.

[9] Welch Labs, JEPA interview / explainer with Yann LeCun, part 1 — https://www.youtube.com/watch?v=kYkIdXwW2AE
Used for LeCun’s claims about JEPA replacing LLMs, blurry generative video prediction, the dashcam/leaves example, and world models.

[10] Welch Labs, JEPA interview / explainer with Yann LeCun, part 2 — https://youtu.be/v_jDvpEGTIg?si=CS9Usu3NoX4J62Gv
Used for LeCun’s VLA critique, action-consequence quote, LeWorldModel explanation, and hierarchy discussion.

[11] Yann LeCun, A Path Towards Autonomous Machine Intelligence, 2022 — https://openreview.net/forum?id=BZ5a1r-kVsf
Used for the common-sense-as-world-models framing.

[12] Shixart1985, Robotic arm at work lifting a box during a technology exhibition, Wikimedia Commons — https://commons.wikimedia.org/wiki/File:Robotic_arm_at_work_lifting_a_box_during_a_technology_exhibition.jpg
CC BY 2.0 source photo used inside the JEPA latent prediction loop illustration.

S Nagaonkar, "JEPA: From Language Models to World Models",
VideoDB Labs, July 2026.

@article{nagaonkar2026jepaLanguageModels,
  author = {S Nagaonkar},
  title = {JEPA: From Language Models to World Models},
  journal = {VideoDB Labs},
  year = {2026},
  note = {https://labs.videodb.io/research/jepa-from-language-models-to-world-models},
}