AI United States

AI Doubling Its Compute Every 3.5 Months

Compute Scoreboard: AlexNet 1 AlphaZero 300,000

AI is booming. So is the compute used to train AI models. San Francisco-based non-profit AI research institute OpenAI yesterday released an analysis on the largest AI models since 2012. Researchers discovered that the amount of compute used for training doubles every 3.5 months. By comparison, the number of transistors per square inch on integrated circuits only doubles every 18 months, a.k.a Moore’s Law.

The analysis also showed that DeepMind’s 2017 AlphaZero, which mastered the game of Go after mere hours of training with no human game records to reference, ran over 300,000 times more compute than AlexNet, the breakthrough GPU-powered convolutional neural network (CNN) that won the 2012 ILSVRC (ImageNet Large-Scale Visual Recognition Challenge).

The results suggest that more compute can achieve better performance in AI models, and is often complementary to algorithmic advances.

The analysis also suggests that the exponential increase of compute for AI training can be largely attributed to hardware development. Before 2012, AI researchers had merely co-opted GPUs for machine learning, but after 2016 specialized processors such as TPUs and more advanced interconnect techniques that enable large-scale parallel computing allowed researchers to train AI models with better compute capability.

OpenAI researchers adopted two methods to measure the compute directly counting operations used in a forward pass, or calculating operations based on numbers of GPUs deployed for training. For example, an AlexNet paper revealed that the network takes “between five and six days to train on two GTX 580 3GB GPUs”, so researchers could estimate an approximate amount of compute of 0.0058 petaflops per day.

Synced collected comments from AI scholars and journalists on OpenAI’s new analysis:

I’m not sure you can conclude that AI capabilities will always advance in step with compute power. That assumes all you need is more data and GPU horsepower.
— Will Knight, MIT Technology Review’s Senior Editor for Artificial Intelligence.

I think it’s totally backwards. Techies love playing with bigger toys, so they’ll try to use whatever they can get their hands on. The chart is showing the increasing availability of resources to DL researchers — that’s all.
— Jeremy Howard, Previously Kaggle President and Chief Scientist, and now Fast.ai Founding Researcher

If there is an underlying technique (architecture search, self-play, …) which can utilize vast compute effectively for a given real task, the amount of compute available for this purpose is astronomical.
— Stephen Merity, Previously Salesforce Senior Research Scientist

This chart via @OpenAI is amazing on its own, but the most interesting part imo is how much of an outlier DQN remains.
— Dave Gershgorn, Quartz Reporter for Artificial Intelligence


Journalist: Tony Peng| Editor: Michael Sarazen

19 comments on “AI Doubling Its Compute Every 3.5 Months

  1. Your article is such an informative article. It is glad to read such those articles thanks for sharing. Akira Hoodie

  2. So AI master games now. What happens when they master humans.
    thanks for this article. comic con costumes for sale

  3. Thanks for this Article its very useful for me.
    Athena Karkanis Manifest S03 Black Jacket

  4. Such a wonderful Article its very good keep it up.
    Manifest Josh Dallas Shirt Style Leather Jacket

  5. I must say this post is really great and informative too, keep sharing more I really like the content. While exploring things if there is any technical issue.

  6. Thank you for posting such a great article! I found your website perfect for my needs.

  7. Are you looking forward to getting your hands on something unique? Visit our website to see our wide range of stylish and fashionable jackets, including Gaming jackets, Biker jackets, Leather jackets, and more. Simply click the link to make a purchase.

  8. Astonishing! What a by and large differentiation to be consolidated here and to be significant for this thoroughly shocking neighborhood people. In case someone let me in on a half year earlier I would be in the position I am today by essentially taking a course called Connect With Anyone, I would never have confided in it

  9. AI very hard site. This AI information benefits people. This is a very nice post.

  10. David Valentine

    I must mention that this post is both interesting and educational. please continue to share as I enjoy the information. Are you excited to get your hands on something special? Then Elvis Austin Butler Leather Jacket is for you. The best leather apparel for your winter collection.

  11. This article is outstanding in every way. Thank you for providing this useful information.
    biker jacket

  12. This article yellowstone jackets is very amazing.Thank for sharing this post.

  13. google great one keep it up

  14. Joan Acosta

    I’m checking out your site for the first time, and I must say that it’s great and very beneficial for kids. You’re doing an outstanding job, and I’ll be returning soon. famous ladies jackets Good luck in the future.

  15. jasonalbert

    Is it accurate to say that you are searching for something special In the event that truly, at that point head to gatekeepers to get the most moving outfits at truly reasonable rates? The nature of the items is very good quality which will show up more appealing at any event and function’ Thanksgiving suit for men

  16. I can’t stop playing Flappy Bird. It makes me happy. Thanks for this great game.

  17. Michelle Evans

    I’m visiting your website for the first time, and I must say that it’s fantastic and very useful for children. You’re doing an excellent job, and I’ll be back shortly. Pelle Pelle Jacket

  18. Alysiajims

    It’s enjoyable to read such lovely and instructive material. I enjoyed reading the useful content on your website. Please keep sharing with us.
    Jack Harlow Varsity Jacket

Leave a Reply to JW william Cancel reply

Your email address will not be published. Required fields are marked *

%d bloggers like this: