AI United States

AI Doubling Its Compute Every 3.5 Months

Compute Scoreboard: AlexNet 1 AlphaZero 300,000

AI is booming. So is the compute used to train AI models. San Francisco-based non-profit AI research institute OpenAI yesterday released an analysis on the largest AI models since 2012. Researchers discovered that the amount of compute used for training doubles every 3.5 months. By comparison, the number of transistors per square inch on integrated circuits only doubles every 18 months, a.k.a Moore’s Law.

The analysis also showed that DeepMind’s 2017 AlphaZero, which mastered the game of Go after mere hours of training with no human game records to reference, ran over 300,000 times more compute than AlexNet, the breakthrough GPU-powered convolutional neural network (CNN) that won the 2012 ILSVRC (ImageNet Large-Scale Visual Recognition Challenge).

The results suggest that more compute can achieve better performance in AI models, and is often complementary to algorithmic advances.

The analysis also suggests that the exponential increase of compute for AI training can be largely attributed to hardware development. Before 2012, AI researchers had merely co-opted GPUs for machine learning, but after 2016 specialized processors such as TPUs and more advanced interconnect techniques that enable large-scale parallel computing allowed researchers to train AI models with better compute capability.

OpenAI researchers adopted two methods to measure the compute directly counting operations used in a forward pass, or calculating operations based on numbers of GPUs deployed for training. For example, an AlexNet paper revealed that the network takes “between five and six days to train on two GTX 580 3GB GPUs”, so researchers could estimate an approximate amount of compute of 0.0058 petaflops per day.

Synced collected comments from AI scholars and journalists on OpenAI’s new analysis:

I’m not sure you can conclude that AI capabilities will always advance in step with compute power. That assumes all you need is more data and GPU horsepower.
— Will Knight, MIT Technology Review’s Senior Editor for Artificial Intelligence.

I think it’s totally backwards. Techies love playing with bigger toys, so they’ll try to use whatever they can get their hands on. The chart is showing the increasing availability of resources to DL researchers — that’s all.
— Jeremy Howard, Previously Kaggle President and Chief Scientist, and now Founding Researcher

If there is an underlying technique (architecture search, self-play, …) which can utilize vast compute effectively for a given real task, the amount of compute available for this purpose is astronomical.
— Stephen Merity, Previously Salesforce Senior Research Scientist

This chart via @OpenAI is amazing on its own, but the most interesting part imo is how much of an outlier DQN remains.
— Dave Gershgorn, Quartz Reporter for Artificial Intelligence

Journalist: Tony Peng| Editor: Michael Sarazen

37 comments on “AI Doubling Its Compute Every 3.5 Months

  1. Your article is such an informative article. It is glad to read such those articles thanks for sharing. Akira Hoodie

  2. So AI master games now. What happens when they master humans.
    thanks for this article. comic con costumes for sale

  3. Thanks for this Article its very useful for me.
    Athena Karkanis Manifest S03 Black Jacket

  4. Such a wonderful Article its very good keep it up.
    Manifest Josh Dallas Shirt Style Leather Jacket

  5. I must say this post is really great and informative too, keep sharing more I really like the content. While exploring things if there is any technical issue.

  6. Thank you for posting such a great article! I found your website perfect for my needs.

  7. Are you looking forward to getting your hands on something unique? Visit our website to see our wide range of stylish and fashionable jackets, including Gaming jackets, Biker jackets, Leather jackets, and more. Simply click the link to make a purchase.

  8. Astonishing! What a by and large differentiation to be consolidated here and to be significant for this thoroughly shocking neighborhood people. In case someone let me in on a half year earlier I would be in the position I am today by essentially taking a course called Connect With Anyone, I would never have confided in it

  9. AI very hard site. This AI information benefits people. This is a very nice post.

  10. David Valentine

    I must mention that this post is both interesting and educational. please continue to share as I enjoy the information. Are you excited to get your hands on something special? Then Elvis Austin Butler Leather Jacket is for you. The best leather apparel for your winter collection.

  11. This article is outstanding in every way. Thank you for providing this useful information.
    biker jacket

  12. This article yellowstone jackets is very amazing.Thank for sharing this post.

  13. google great one keep it up

  14. Joan Acosta

    I’m checking out your site for the first time, and I must say that it’s great and very beneficial for kids. You’re doing an outstanding job, and I’ll be returning soon. famous ladies jackets Good luck in the future.

  15. jasonalbert

    Is it accurate to say that you are searching for something special In the event that truly, at that point head to gatekeepers to get the most moving outfits at truly reasonable rates? The nature of the items is very good quality which will show up more appealing at any event and function’ Thanksgiving suit for men

  16. I can’t stop playing Flappy Bird. It makes me happy. Thanks for this great game.

  17. Michelle Evans

    I’m visiting your website for the first time, and I must say that it’s fantastic and very useful for children. You’re doing an excellent job, and I’ll be back shortly. Pelle Pelle Jacket

  18. Alysiajims

    It’s enjoyable to read such lovely and instructive material. I enjoyed reading the useful content on your website. Please keep sharing with us.
    Jack Harlow Varsity Jacket

  19. Trilaen

    Great Blog. The Best Wi-Fi SMS app is TextNow, which allows users to send unlimited text messages over Wi-Fi for free. It also includes features like group messaging, voice messaging, and multimedia sharing.

  20. Natasha

    I enjoy your blog. Great blog, keep sharing this type of blog again I visit this type of blog. I share this Poker Face Natasha Lyonne Brown Leather Jacket is constructed of real leather, and the viscose lining inside provides warmth and comfort.

  21. Bruce Wayne

    I’m visiting this website for the first time I’m very happy to say that your blog is wonderful it’s a lot of time I spend on this website I will come again to this website.

  22. Mike Rooney

    This is really a great website, i suggest you to post articles to attract visitors attention. Your website is really a great source of information.Snoop Dogg Super Bowl Halftime Tracksuit

  23. Alex Mercer

    I appreciate you sharing this fantastic article. I adore your article. This website is very user friendly and well done. Thank you for contributing your ideas. Jensen Ackles Jacket Please feel free to browse my web.

  24. Andrew Mark

    Wonderful article. Fascinating to read. I love to read such an excellent article. Thanks! It has made my task more and extra easy. Keep rocking.
    Yellowstone Merch

  25. Peter Parker

    I appreciate the research and effort you put into creating this content.
    Walter White Jacket

  26. Jennifer Connelly

    Your ideas and info is really unique and you clear my all doubts about this topic. I also want to share some thing unique try this Attack On Titan Eren Jeager Cosplay Black Hooded Coat

  27. christome

    Those who grew up with their favourite media franchises will love the touch of celebrity attire, such as this
    Half Shark Camo BAPE Hoodie worn by TV actress, singer. Jacket of a famous person

  28. Daniel Alexander

    AI has come a long way in the era that is going on at the moment which is also helping a lot of people. I also have to write content about a Star Trek Picard Season 3 Deanna Troi Leather Jacket for my siteso I take the help of ai. And I enjoy reading your ai related content.

  29. chrisjimmy

    I searched a lot about AI on the net, but I did not find any content like yours. AI is very useful, so everyone searches about AI on the net. Now I will give you some fashion information regarding winter. In this winter season, I will advise you to use this And Just Like That S02 Carrie Bradshaw Plaid Coat

  30. It is the best time to upgrade your look and feel free to get this amazing Johnny Lawrence Red Cobra Kai Jacket. the leather-fabricated outerwear is just best to make you stand out in any event.

  31. Smoke wholesale Texas refers to the thriving business of supplying various smoking products and accessories in the state of Texas. Texas, known for its rich barbecue culture and appreciation for smoking meats, has a robust market for wholesale smoking products. Wholesalers in Texas offer a wide range of items, including premium cigars, tobacco, rolling papers, hookahs, and vaping supplies.

  32. jason statham

    Wow its a very good post.
    NewYork Jacket

  33. jhonny bravo

    Very nice Article Keep it up Squid Game Season 2 Tracksuit

  34. kenna james

    Keep sharing good information
    Superheroes Costumes

  35. mollyharboe

    You are absolutely right that AI is on the rise. And if AI is used for a specific purpose, that’s something we can use. Like the Karlie Kloss Red Bomber Jacket has a special purpose that makes you look beautiful and fashionable.

  36. Tina Anderson

    AI is the biggest and most advanced technology of this era. And indeed AI is on the rise. And if AI is used in a positive way, it can be very good. Now I will suggest you to use this A Royal Christmas Crush 2023 Stephen Huszar Puffer Coat in this cold season.

Leave a Reply

Your email address will not be published. Required fields are marked *