AI Industry Machine Learning & Data Science Nature Language Tech Research

AI21 Labs Asks: How Much Does It Cost to Train NLP Models?

How much is this going to cost? And what are the main factors affecting that price tag?

These days the total cost of training an NLP model can climb into millions of dollars. And so it is only natural that budget-constrained researchers, engineers and scientists, when planning their model-training experiments, ask the important question: How much is this going to cost? And what are the main factors affecting that price tag?

Israeli research company AI21 Labs looks for answers in their recently published paper The Cost of Training NLP Models: A Concise Overview.

image.png

AI21 Labs Co-CEO, Stanford University Professor of Computer Science (emeritus), and AI Index initiator Yoav Shoham describes the motivation for the project. “It started with an inquiry we got at the AI Index. I started jotting down a quick answer and realized it deserved a longer one. I also realized we had a lot of the expertise at AI21 Labs. So we spun up a small effort to put this report together, to benefit the community.”

The team compared three different-sized Google BERT language models on the 15 GB Wikipedia and Book corpora, evaluating both the cost of a single training run and a typical, fully-loaded model cost. The team estimated fully-loaded cost to include hyperparameter tuning and multiple runs for each setting: “We look at a somewhat modest upper bound of two configurations and ten runs per configuration.”

  • $2.5k – $50k (110 million parameter model)
  • $10k – $200k (340 million parameter model)
  • $80k – $1.6m (1.5 billion parameter model)

Training costs can vary drastically due to different technical parameters, climbing up to US$1.3 million for a single run when training Google’s 11 billion parameter Text-to-Text Transfer Transformer (T5) neural network model variant. A project that might require several runs could see total training costs hit a jaw-dropping US$10 million.

The researchers note that the cost for floating-point operations (FLOPs) and basic neural network operation for example have been falling, while the rise in overall costs is being driven by increasing dataset and model size and training volume: “In NLP, everything is big and getting bigger.”

image.png

Precise cost estimates based on particular NLP models or task training procedures are however difficult. As Shoham told Synced “It’s influenced by multiple parameters. Besides the technical parameters, there are also personal and organizational considerations; a researcher might be impatient to wait three weeks to do a thorough analysis, and his/her organization may not be able to, or wish to pay for it. So for the same task one could spend $100k or $1 million.”

The study says the reason for NLP’s current mega-scale, brute-force statistical approach and SOTA leaderboard races is simple: “It works; it has yielded better performance than any alternative.”

Shoham suggests however that it’s time for a change, as chasing leaderboard rankings on narrow tasks often leads to over-optimizing for the challenge set and overfitting. “The community knows this,” he says.

Instead of devoting more compute to leaderboard races, Shoham proposes the research community look into more efficient neural network architectures and how to take better advantage of data.

The paper The Cost of Training NLP Models: A Concise Overview is on arXiv.


Journalist: Fangyu Cai | Editor: Michael Sarazen

45 comments on “AI21 Labs Asks: How Much Does It Cost to Train NLP Models?

  1. Pingback: Google Trains a Trillion-Parameter Language Model

  2. Pingback: In the cloud — Sparsity on GPUs provides 5X speedup – Ramsey Elbasheer | History & ML

  3. Pingback: In the cloud — Sparsity on GPUs provides 5X speedup | NEO Share

  4. Pingback: In the cloud — Sparsity on GPUs provides 5X speedup – EDGENeural.ai

  5. Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - Techio

  6. Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - Modern Future Tech

  7. Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions | VentureBeat

  8. Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - Tech Insider Updates

  9. Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - EZ Tech Profits

  10. Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - Technology Streets

  11. Pingback: AI Weekly: AI model training costs are rising, underscoring the need for new solutions | MCUTimes

  12. Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions – Oracle Dætalytica

  13. Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - venturecurrent.com

  14. Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions – TOP TECH

  15. Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - Social News Hubb

  16. Pingback: AI Weekly: AI mannequin practising charges on the upward thrust, highlighting want for contemporary choices — UsaDots – UsaDots

  17. Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - Lowballdealz

  18. Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - Science News

  19. Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - Tech News

  20. Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - Top Tech News

  21. Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions – TECHOSMO

  22. Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - Futurist Journal

  23. Pingback: AI Weekly: AI version training expenses growing, highlighting demand for brand-new remedies - Technology Life

  24. Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - Town Celeb

  25. Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - Luxuryfolks

  26. Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - Washington Dailies

  27. Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions – Adfero News

  28. Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - RockingTimes

  29. Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - Health Astros

  30. Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions • CAPTIS

  31. Pingback: AI Weekly: AI mannequin coaching prices on the rise, highlighting want for brand new options » Best IND News

  32. Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - sfgtechnology.com

  33. Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - thecowbotchannel.com

  34. Pingback: AI Weekly: los costos de entrenamiento del modelo de AI aumentan, lo que destaca la necesidad de nuevas soluciones - Blog | Internet, Cibersegurdad, Digitalización y Tecnología

  35. Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions – 365ViralBuzz

  36. Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - The News Global

  37. Pingback: AI Weekly: AI mannequin coaching prices on the rise, highlighting want for brand new options – Kabar kabari

  38. Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - Cazy Boy Tech

  39. Pingback: AI Weekly: AI mannequin coaching prices on the rise, highlighting want for brand spanking new options – Kilden Info

  40. Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions – Omni Innotech

  41. Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions | TechieTricks.com

  42. Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - AI Caosuo

  43. Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions – Eco Tech Digital

  44. Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions – VentureBeat - AI Caosuo

  45. Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - epicapplications

Leave a Reply

Your email address will not be published.

%d bloggers like this: