> ## Documentation Index
> Fetch the complete documentation index at: https://aegean.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# World Models
> Learning internal environment models for planning, control, and embodied reasoning, from Ha & Schmidhuber's original architecture to modern diffusion and autoregressive variants.
Humans build an internal mental model of the world and routinely use simulation to update their expectations. When you catch a ball, you are not reacting to each frame of visual input, you are predicting the ball's trajectory in a learned internal model and planning your arm's motion against that prediction.
World models bring this idea to machine learning: instead of learning a policy directly from environment interactions (model-free RL), the agent first learns a compressed, predictive model of the environment and then trains a controller *inside* that model. The agent literally learns to act inside its own dream.
## What a world model is
At its most general, a world model is any learned function
```
(observation history, action) → next observation (or distribution over next observations)
```
that the agent can roll out forward in time without touching the real environment. Once you have such a function, you can:
* **Plan** by searching over hypothetical action sequences
* **Train a policy** entirely on imagined rollouts (no real-world samples needed)
* **Estimate uncertainty** by sampling multiple futures from the model
This is the deep-learning incarnation of [model-based reinforcement learning](/aiml-common/lectures/reinforcement-learning/model-based-algorithms/index). Classical model-based RL hand-coded the transition function (Dyna, MPC); modern world models learn it directly from pixels.
## A family of architectures
World models have evolved through several generative-modeling backbones. Each generation kept the core idea, learn an internal simulator, train a controller inside it, but improved the fidelity and scope of the dream.
| Generation | Vision backbone | Example | Strength |
| ---------------------------------------------------------------------- | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
| **[Latent VAE + RNN](/book/world-models/seminal-model/index)** | VAE | [Ha & Schmidhuber (2018)](/book/world-models/seminal-model/index) | Clean decomposition, easy to train, low compute |
| **Recurrent latent dynamics** | VAE | [Dreamer / DreamerV3](https://arxiv.org/abs/2301.04104) | Single architecture works across Atari, DMControl, Minecraft |
| **Diffusion-based** | Diffusion model | [DIAMOND](https://arxiv.org/abs/2405.12399), [Genie 2](https://deepmind.google/discover/blog/genie-2-a-large-scale-foundation-world-model/) | Photorealistic next-frame prediction, the dream looks like reality |
| **Autoregressive token** | Transformer | [GAIA-1](https://arxiv.org/abs/2309.17080) | Multi-modal conditioning, supports text-conditioned scenario generation |
| **[Joint-Embedding Predictive (JEPA)](/book/world-models/jepa/index)** | ViT (JEPA) | [I-JEPA](https://arxiv.org/abs/2301.08243), [V-JEPA](https://arxiv.org/abs/2404.08471) | Predicts in representation space, ignores irrelevant pixel detail, scales to video and language |
The original VAE-based architecture has blurry reconstructions and a fixed latent dimensionality, but it remains the cleanest place to build intuition before tackling the larger variants. Diffusion and autoregressive models achieve dramatically higher visual fidelity at the cost of much larger parameter counts and slower rollouts. The [JEPA family](/book/world-models/jepa/index) takes a different approach entirely: instead of reconstructing pixels or tokens, it predicts *representations* of future observations, letting the model focus on semantically meaningful structure.
## World models vs other approaches
It helps to put world models next to the alternatives you have already seen for learning a control policy. Each approach has a different failure mode:
| Approach | How it learns | Failure mode |
| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- |
| **Behavioral cloning** | Supervised regression on expert demonstrations (where the expert is itself produced by an RL algorithm such as PPO, or by human teleoperation) | Distribution shift, compounds errors on unseen states |
| **World model** | Learn a model of the environment, then train a controller in the dream | Model inaccuracy, the dream may diverge from reality |
## How world models connect to the rest of Physical AI
World models sit at the intersection of several threads in this course:
**Sim-to-real.** A world model trained on real data *is* a simulator, one that is automatically calibrated to reality because it was learned from real observations. This eliminates the hand-authored sim gap discussed in the [sim-to-real transfer](/aiml-common/lectures/sim2real/index) page. Instead of building Gazebo worlds and hoping they match reality, you learn a world model from a few minutes of real video and train inside it.
**Behavioral cloning vs world models.** [BC](/aiml-common/lectures/imitation-learning/behavioral-cloning/index) treats the expert's demonstrations as the only source of data. World models treat the environment's dynamics as learnable, the agent can extrapolate beyond the demonstrations by imagining what would happen in unseen states. This is why world models partially address BC's distribution-shift problem: the agent has dreamed about a wider distribution of states than any fixed demonstration dataset provides.
**VLA models.** Most current [VLA architectures](/book/vla-agents/index) like [OpenVLA](/book/vla-agents/openvla/index) and [RT-2](https://robotics-transformer2.github.io/) are pure behavior-cloning systems with no explicit world model. Combining them with learned dynamics models is an active research direction precisely because doing so could close the BC-vs-world-model gap above.
## In this section
Ha & Schmidhuber's original V/M/C architecture, VAE for vision, MDN-RNN for memory, tiny linear controller trained inside the dream.
LeCun's roadmap for world models, predict *representations*, not pixels. From the 2022 position paper through I-JEPA, MC-JEPA, V-JEPA, VL-JEPA, H-JEPA, and LeJEPA.
## Further reading
* Ha & Schmidhuber (2018). [World Models](https://worldmodels.github.io/), the seminal paper
* Hafner et al. (2020). [Dream to Control: Learning Behaviors by Latent Imagination](https://arxiv.org/abs/1912.01603), Dreamer, with actor-critic training in latent space
* Hafner et al. (2023). [Mastering Diverse Domains through World Models](https://arxiv.org/abs/2301.04104), DreamerV3, the same architecture across Atari, DMControl, and Minecraft
* Alonso et al. (2024). [Diffusion for World Modeling: Visual Details Matter in Atari](https://arxiv.org/abs/2405.12399), DIAMOND
* Hu et al. (2023). [GAIA-1: A Generative World Model for Autonomous Driving](https://arxiv.org/abs/2309.17080)
***
[Edit this page on GitHub](https://github.com/aegean-ai/eaia/edit/main/src/book/world-models/index.mdx) or [file an issue](https://github.com/aegean-ai/eaia/issues/new/choose).