In 2012, Geoffrey Hinton’s research team used only two NVIDIA GPUs to train AlexNet, the revolutionary network architecture that handily won the ImageNet Large Scale Visual Recognition Challenge. It probably never occurred to these groundbreaking researchers that just seven years later, a new team of researchers would use almost 10,000 times more GPUs to train their AI model.
A research team from NVIDIA, Oak Ridge National Laboratory (ORNL), and Uber has introduced new techniques that enabled them to train a fully convolutional neural network on the world’s fastest supercomputer, Summit, with up to 27,600 NVIDIA GPUs. They managed to achieve an impressive, near-linear scaling of 0.93 on distributed training and produce a model capable of atomically-accurate reconstruction of materials — a longstanding scientific problem involving materials imaging.
What is Summit? In June 2018 the US Department of Energy’s Oak Ridge National Laboratory in Tennessee unveiled the world’s fastest supercomputer Summit, whosecomputing power reaches 200 petaflops. Summit employs over 9,000 general-purpose IBM processors and 28,000 NVIDIA graphics processors along with state-of-the-art interconnect, which makes it very well-suited for deep learning workloads.
Why does this research matter? The growing demand for AI compute is driving the development of distributed machine learning to multiple processors, particularly a dataparallelism technique that distributes data across different nodes to process in parallel.
Most existing training techniques are tailored for scenarios that require only tens to hundreds of AI processors. These methods however fail to scale efficiently on supercomputers with tens of thousands of AI accelerators. That has motivated scientists to explore data parallelism problems on supercomputers.
Major contributions: Researchers proposed new gradient reduction strategies which produce optimal overlap between computation and communication. Essentially these new strategies comprise two techniques: a lightweight worker coordination technique (BitAllReduce) and a gradient tensor grouping strategy (Grouping). Researchers say these two orchestration strategies “improve on different aspects of distributed deep learning as currently implemented in Horovod.”
In their experiments, researchers achieved a scaling efficiency of 0.93 at 4600 nodes (27,600 GPUs) during distributed deep learning. In comparison, using 60 NVIDIA K80 GPUs to train ResNet-152 only obtains a 50X speed-up — equivalent to a scaling efficiency of 0.83, according to the benchmark test on TensorFlow.
Researchers measured power consumption — an indicator of the efficiency on a supercomputer — of the main hardware components on Summit during distributed training, and also profiled the compute performance for distributed training with the proposed strategies based on I/O, computation performed for the DNN forward and backward propagation, and communication operations embedded in the computation graph.
Researchers also applied a deep neural network to solve the phase problem in the atomic imaging of materials, using a dataset with size of a 500 terabytes.
Implications: Nouamane Laanait, the paper’s first author, told Synced that “the novel distributed deep learning techniques we introduced both increase the training efficiency of existing deep learning (DL) applications but also open up new frontiers in developing DL models in new domains, for instance the physical sciences, whose massive data sizes necessitate the use of high-performance computing. Scaling up models and data sizes has in the past advanced the field of machine learning, our work is very much in that spirit, where we efficiently scaled-up distributed training to the computing scale of the world’s fastest supercomputer.”
Related news: NVIDIA announced another scaling breakthrough in August — training the BERT-Large language model in just 53 minutes using 1,472 V100 GPUs GPUs. The NVIDIA research group also managed to scale the training of the GPU-2 model 8.3 billion parameters, using 8-way model parallelism and 64-way data parallelism on 512 GPUs.
The paper Exascale Deep Learning for Scientific Inverse Problems is on arXiv.
Journalist: Tony Peng | Editor: Michael Sarazen
Fantastic beat ! I would like to apprentice at the same time as you amend
your website, how can i subscribe for a weblog website?
The account helped me a applicable deal. I have been a little bit familiar
of this your broadcast offered shiny clear idea
I really like and appreciate your article post. You amazingly come with superb articles and much helpful for any industry as well as for people. Thanks for sharing this info Keep on working, great Work!
bigpond login
Thanks for sharing all this information here on this page.
Nice This Post Very Informatic
Very nice blog. Thanks
It’s amazing to see how far we’ve come since AlexNet — going from two GPUs to nearly 28,000 in just seven years is a staggering leap. The near-linear scaling of 0.93 on Summit is particularly impressive because it shows that distributed training can actually remain efficient at such a massive scale, which is often the biggest hurdle in supercomputing AI. The application to atomically-accurate reconstruction of materials is also a great example of how AI can push the boundaries of scientific discovery. For anyone interested in the intersection of AI and scientific computing, this kind of work really highlights what’s possible when you pair advanced algorithms with world-class hardware. If you’re looking to explore more about AI applications and tools, I’d recommend checking out The RoboLibrary Homepage, which has a lot of useful resources for researchers and practitioners alike. Thanks for sharing this fascinating update!
This is a fascinating leap from the two GPUs used for AlexNet to 27,600 on Summit. The near-linear scaling of 0.93 is particularly impressive—distributed training often suffers from communication overhead, so achieving that efficiency at this scale speaks to the quality of the techniques developed. The application to atomically-accurate reconstruction of materials is a great reminder that these massive compute feats ultimately serve real scientific breakthroughs. It also makes you wonder how accessible such methods will become for smaller research groups. As we push the limits of scale, sometimes we need clever tools to manage complexity—similar to how a gothic lockpick calculator helps players streamline a different kind of intricate challenge. Great write-up on a milestone worth watching.
This is a fascinating milestone in distributed deep learning. The jump from AlexNet’s two GPUs to 27,600 on Summit really puts the pace of innovation into perspective, and the near-linear scaling of 0.93 is especially impressive for such a massive system. It’s also great to see the materials science payoff — atomically-accurate reconstruction is exactly the kind of complex problem that needs this level of compute. As training demands keep growing, it feels like we’re only scratching the surface of what’s possible with supercomputing-scale AI. On a related note, if you enjoy exploring ambitious, boundary-pushing technologies, you might also enjoy a different kind of world-building in the fourth wing series books. Thanks for sharing this research — looking forward to seeing where these techniques lead next.
This is a fascinating milestone. Going from two GPUs in 2012 to 27,600 on Summit shows just how far distributed deep learning has come. Achieving 0.93 scaling efficiency at that scale is no small feat, and it’s great to see supercomputing applied to materials science in such a practical way. The atomic-level reconstruction problem has been a tough one for a long time, so this could open up serious new avenues for material design and discovery. Thanks for breaking down the technical details so clearly. On a lighter note, for anyone who wants to jazz up their own posts or comments about AI research, I came across this handy tool for useful text styling options: letras diferentes para copiar e colar – not quite as powerful as Summit, but fun for a custom look. Looking forward to seeing where this scaling research heads next.
It’s remarkable to see how far large-scale deep learning has come since AlexNet’s two-GPU days. Achieving near-linear scaling across 27,600 GPUs on Summit is no small feat, and applying that power to atomically-accurate materials reconstruction shows just how impactful distributed training can be for real scientific breakthroughs. I’d be curious to know how the team handled communication overhead and fault tolerance at that scale. On a lighter note, after reading about such intense compute challenges, it’s nice to take a mental break now and then — I actually stumbled across a fun guide for casual gameplay at cách chơi game trà sữa when I needed a distraction. Thanks for sharing this fascinating update on the state of AI research!
This is a remarkable demonstration of how far distributed deep learning has come—from AlexNet on two GPUs to near-linear scaling across 27,600. Achieving a scaling efficiency of 0.93 at that scale is no small feat, and applying it to atomically-accurate material reconstruction shows the tangible scientific payoff of such engineering work. It’s exciting to think what other long-standing problems could be tackled as these techniques mature.
On a related note, just as we modernize AI infrastructure, it’s also important to preserve and convert legacy digital content. For anyone working with older Bengali texts, I’ve found পুরনো বিজয় লেখা ইউনিকোডে রূপান্তর quite helpful for bridging that gap. Hopefully we’ll see similar efforts to keep all kinds of data accessible as computational tools evolve.
This is a remarkable leap from the two GPUs used in the AlexNet days to 27,600 on Summit. Achieving near-linear scaling of 0.93 at that scale is no small feat—it really shows how far distributed training techniques have come. The ability to reconstruct materials with atomic accuracy could open up huge possibilities in materials science and beyond. It’s also fascinating to see how these massive advances in computing eventually cascade down into more everyday technologies—even something as simple as a unicode to bijoy convert utility benefits from the broader progress in efficient processing. Congratulations to the research team on this milestone; I’m excited to see what they tackle next.