NYU & Stanford’s GPUDrive: Achieving Over 1 Million Steps per Second in Multi-Agent Driving Simulations

Multi-agent learning algorithms have excelled at producing superhuman planning abilities in various games, but their impact on real-world multi-agent planning systems remains limited. A major barrier is the massive data requirements—often billions of experience steps—needed for effective training.

In a new paper GPUDrive: Data-driven, multi-agent driving simulation at 1 million FPS, a research team from New York University and Stanford University presents GPUDrive, a GPU-accelerated multi-agent simulator built on the Madrona Game Engine. GPUDrive is capable of generating over a million experience steps per second, making it a game-changer for applying sample-inefficient yet powerful reinforcement learning (RL) algorithms to multi-agent planner design.

GPUDrive is designed to combine real-world driving data with unprecedented simulation speeds. Running at over one million steps per second on both consumer-grade and datacenter-class GPUs, it maintains a minimal memory footprint, allowing for the simulation of hundreds to thousands of environments simultaneously, each with hundreds of agents. This scalability enables researchers to train RL models efficiently while keeping the computational load manageable.

The simulator supports a wide range of sensor modalities, including LIDAR and human-like visual fields, making it highly versatile for studying the impact of different sensor inputs on agent behavior. GPUDrive operates in three distinct modes. The first mode provides complete observability within a fixed radius, primarily for debugging and rapid testing. The other two modes simulate real-world sensor limitations: one mimics what an autonomous vehicle would perceive via LIDAR, and the other replicates a human’s visual perspective.

GPUDrive also incorporates driving logs and maps from established self-driving datasets, facilitating the integration of both imitation learning and reinforcement learning techniques. This allows researchers to explore the development of autonomous vehicles alongside the modeling of human behaviors such as driving, cycling, and walking.

Empirical results show that GPUDrive is highly effective at training RL agents across numerous scenes in the Waymo Motion dataset. Agents are able to achieve strong goal-reaching performance within minutes for individual scenes and exhibit generalized planning capabilities within a few hours.

This work represents a significant advancement in scaling reinforcement learning for multi-agent planning in complex, safety-critical environments that involve both humans and autonomous systems. The research team has also released implementations of optimized RL algorithms that can process 20 million experience steps per hour on consumer-grade GPUs. In addition, they open-sourced high-performance driving agents that achieve a 97% success rate on a subset of training scenes, providing a strong foundation for future research in this area.

The code is available on project’s GitHub. The paper GPUDrive: Data-driven, multi-agent driving simulation at 1 million FPS is on arXiv.

---

Author: Hecate He | Editor: Chain Zhang