Yann LeCun Team’s New Research: Revolutionizing Visual Navigation with Navigation World Models

Navigation is a fundamental skill for any visually-capable organism, serving as a critical tool for survival. It enables agents to locate resources, find shelter, and avoid threats. In humans, navigation often involves mentally simulating possible future paths while accounting for constraints and alternative possibilities. However, modern robotic navigation systems are far less flexible. Current state-of-the-art navigation policies are typically “hard-coded,” meaning once training is complete, introducing new constraints is difficult. Furthermore, existing supervised visual navigation models struggle to allocate additional computational resources when facing more complex navigation tasks.

To address the abovementioned issues, in a new paper Navigation World Models, a research team from Meta, New York University and Berkeley AI Research proposes a Navigation World Model (NWM), a controllable video generation model designed to predict future visual observations based on past observations and navigation actions. This model enables agents to simulate potential navigation plans and assess their feasibility before taking action.

NWM is trained using a large dataset of video footage and navigation actions collected from various robotic agents. The model learns to predict the future representations of video frames, given the representations of past frames and corresponding navigation actions. After training, NWM can plan navigation trajectories in new environments by simulating potential paths and verifying if they lead to the target destination.

Conceptually, NWM draws inspiration from recent diffusion-based world models, such as DIAMOND and GameNGen, which are used for offline model-based reinforcement learning. However, unlike these models, NWM is trained on a wide range of environments and agent embodiments. By leveraging this diverse dataset, the researchers successfully trained a large diffusion transformer model that can generalize across multiple environments. This generalization capability is a significant departure from previous models that are often constrained to specific environments or tasks.

NWM also shares conceptual similarities with Novel View Synthesis (NVS) methods like NeRF and GDC. However, while NVS methods aim to reconstruct 3D scenes from 2D images, NWM’s objective is more ambitious: it seeks to train a single model capable of navigating across diverse environments. Unlike NVS approaches, NWM does not rely on 3D priors but instead models temporal dynamics directly from natural video data.

A key technical component of NWM is the Conditional Diffusion Transformer (CDiT), which predicts the next visual state given past image states and actions as input. Unlike a standard Diffusion Transformer (DiT), CDiT offers significantly better computational efficiency. Its complexity scales linearly with the number of context frames, allowing it to handle larger models with up to 1 billion parameters across diverse environments and agent embodiments. This efficiency allows CDiT to require four times fewer FLOPs than a standard DiT, all while delivering superior future prediction results.

The research team conducted extensive experiments to validate NWM’s capabilities. One notable experiment involved using NWM in unfamiliar environments, where it benefited from training on unlabeled, action-free, and reward-free video data from the Ego4D dataset. Qualitatively, NWM demonstrated improved video prediction and generation on individual images. Quantitatively, it achieved more accurate future predictions on the Stanford Go dataset when trained with additional unlabeled video data. These results highlight NWM’s ability to generalize effectively across unseen environments, a key advantage for real-world navigation tasks.

In summary, the Navigation World Model (NWM) represents a powerful leap forward for robotic navigation. Its ability to simulate, plan, and adapt to new constraints makes it a promising approach for building more autonomous and flexible robotic systems.

The project page is available here. The paper Navigation World Models is on arXiv.

---

Author: Hecate He | Editor: Chain Zhang

---