AI Technology & Industry Review
A research team from ETH Zurich and NVIDIA proposes a training framework that enables fast policy generation for real-world robotic tasks — training time can be reduced by multiple orders of magnitude using massive parallelism on a single workstation GPU.
A research team from Google proposes GSPMD, an automatic parallelism system for ML computation graphs that uses simple tensor sharding annotations to achieve different parallelism paradigms in a unified way, including data parallelism, within-layer model parallelism, spatial partitioning, weight-update sharding, optimizer-state sharding and pipeline parallelism.
A research team from NVIDIA, Stanford University and Microsoft Research propose a novel pipeline parallelism approach that improves throughput by more than 10 percent with a comparable memory footprint, showing such strategies can achieve high aggregate throughput while training models with up to a trillion parameters.
“We want to be able to train a machine learning model, like Google Translate, in a few hours,” said Prof. Hwu. Best known for enabling the parallelism capability of the GPU to solve scientific problems like AI.