Tag: parallel computing

AI Machine Learning & Data Science Research

ETH Zurich and NVIDIA’s Massively Parallel Deep RL Enables Robots to Learn to Walk in Minutes

A research team from ETH Zurich and NVIDIA proposes a training framework that enables fast policy generation for real-world robotic tasks — training time can be reduced by multiple orders of magnitude using massive parallelism on a single workstation GPU.

AI Machine Learning & Data Science Research

Google Presents New Parallelization Paradigm GSPMD for common ML Computation Graphs: Constant Compilation time with Increasing Devices

A research team from Google proposes GSPMD, an automatic parallelism system for ML computation graphs that uses simple tensor sharding annotations to achieve different parallelism paradigms in a unified way, including data parallelism, within-layer model parallelism, spatial partitioning, weight-update sharding, optimizer-state sharding and pipeline parallelism.

AI Machine Learning & Data Science Popular Research

NVIDIA, Stanford & Microsoft Propose Efficient Trillion-Parameter Language Model Training on GPU Clusters

A research team from NVIDIA, Stanford University and Microsoft Research propose a novel pipeline parallelism approach that improves throughput by more than 10 percent with a comparable memory footprint, showing such strategies can achieve high aggregate throughput while training models with up to a trillion parameters.