parallel computing

by Synced 2021-05-17 0

Google Presents New Parallelization Paradigm GSPMD for common ML Computation Graphs: Constant Compilation time with Increasing Devices

A research team from Google proposes GSPMD, an automatic parallelism system for ML computation graphs that uses simple tensor sharding annotations to achieve different parallelism paradigms in a unified way, including data parallelism, within-layer model parallelism, spatial partitioning, weight-update sharding, optimizer-state sharding and pipeline parallelism.

by Synced 2021-04-15 6

AI Machine Learning & Data Science Popular Research

NVIDIA, Stanford & Microsoft Propose Efficient Trillion-Parameter Language Model Training on GPU Clusters

A research team from NVIDIA, Stanford University and Microsoft Research propose a novel pipeline parallelism approach that improves throughput by more than 10 percent with a comparable memory footprint, showing such strategies can achieve high aggregate throughput while training models with up to a trillion parameters.

by Synced 2017-05-04 0

China Chinese AI Talent in US Community United States

Make AI Computing 100 Times Faster

“We want to be able to train a machine learning model, like Google Translate, in a few hours,” said Prof. Hwu. Best known for enabling the parallelism capability of the GPU to solve scientific problems like AI.