Paper Survey : World Models

I put together a slide deck surveying "World Models" (Ha & Schmidhuber, 2018), one of the more influential papers in model-based reinforcement learning. The core idea: instead of training an agent directly on raw environment interactions, separate it into three components: • V (Vision) — a convolutional VAE that compresses each frame into a compact latent vector • M (Memory) — an MDN-RNN that models the probability distribution of future latent states • C (Controller) — a minimal linear layer trained with CMA-ES on top of V and M Two key results from the paper: 1. The agent was the first to solve CarRacing-v0 (score 906 ± 21), working directly from raw pixels 2. On VizDoom Take Cover, the controller was trained entirely inside a hallucinated environment — no real game engine during training — and transferred successfully to the actual game One finding worth noting: a controller trained inside an imperfect world model will exploit its blind spots. The authors address this by increasing the temperature of the MDN-RNN, making the dream harder and forcing the agent to learn more robust strategies. The slides also cover the architecture in detail, the historical roots of the idea going back to Schmidhuber (1990), and the current limitations around VAE feature selection and world model capacity. The original paper and interactive demo are available at worldmodels.github.io #ReinforcementLearning #MachineLearning #DeepLearning