These days the total cost of training an NLP model can climb into millions of dollars. And so it is only natural that budget-constrained researchers, engineers and scientists, when planning their model-training experiments, ask the important question: How much is this going to cost? And what are the main factors affecting that price tag?
Israeli research company AI21 Labs looks for answers in their recently published paper The Cost of Training NLP Models: A Concise Overview.

AI21 Labs Co-CEO, Stanford University Professor of Computer Science (emeritus), and AI Index initiator Yoav Shoham describes the motivation for the project. “It started with an inquiry we got at the AI Index. I started jotting down a quick answer and realized it deserved a longer one. I also realized we had a lot of the expertise at AI21 Labs. So we spun up a small effort to put this report together, to benefit the community.”
The team compared three different-sized Google BERT language models on the 15 GB Wikipedia and Book corpora, evaluating both the cost of a single training run and a typical, fully-loaded model cost. The team estimated fully-loaded cost to include hyperparameter tuning and multiple runs for each setting: “We look at a somewhat modest upper bound of two configurations and ten runs per configuration.”
- $2.5k – $50k (110 million parameter model)
- $10k – $200k (340 million parameter model)
- $80k – $1.6m (1.5 billion parameter model)
Training costs can vary drastically due to different technical parameters, climbing up to US$1.3 million for a single run when training Google’s 11 billion parameter Text-to-Text Transfer Transformer (T5) neural network model variant. A project that might require several runs could see total training costs hit a jaw-dropping US$10 million.
The researchers note that the cost for floating-point operations (FLOPs) and basic neural network operation for example have been falling, while the rise in overall costs is being driven by increasing dataset and model size and training volume: “In NLP, everything is big and getting bigger.”

Precise cost estimates based on particular NLP models or task training procedures are however difficult. As Shoham told Synced “It’s influenced by multiple parameters. Besides the technical parameters, there are also personal and organizational considerations; a researcher might be impatient to wait three weeks to do a thorough analysis, and his/her organization may not be able to, or wish to pay for it. So for the same task one could spend $100k or $1 million.”
The study says the reason for NLP’s current mega-scale, brute-force statistical approach and SOTA leaderboard races is simple: “It works; it has yielded better performance than any alternative.”
Shoham suggests however that it’s time for a change, as chasing leaderboard rankings on narrow tasks often leads to over-optimizing for the challenge set and overfitting. “The community knows this,” he says.
Instead of devoting more compute to leaderboard races, Shoham proposes the research community look into more efficient neural network architectures and how to take better advantage of data.
The paper The Cost of Training NLP Models: A Concise Overview is on arXiv.
Journalist: Fangyu Cai | Editor: Michael Sarazen
Pingback: Google Trains a Trillion-Parameter Language Model
Pingback: In the cloud — Sparsity on GPUs provides 5X speedup – Ramsey Elbasheer | History & ML
Pingback: In the cloud — Sparsity on GPUs provides 5X speedup | NEO Share
Pingback: In the cloud — Sparsity on GPUs provides 5X speedup – EDGENeural.ai
Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - Techio
Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - Modern Future Tech
Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions | VentureBeat
Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - Tech Insider Updates
Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - EZ Tech Profits
Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - Technology Streets
Pingback: AI Weekly: AI model training costs are rising, underscoring the need for new solutions | MCUTimes
Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions – Oracle Dætalytica
Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - venturecurrent.com
Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions – TOP TECH
Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - Social News Hubb
Pingback: AI Weekly: AI mannequin practising charges on the upward thrust, highlighting want for contemporary choices — UsaDots – UsaDots
Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - Lowballdealz
Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - Science News
Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - Tech News
Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - Top Tech News
Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions – TECHOSMO
Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - Futurist Journal
Pingback: AI Weekly: AI version training expenses growing, highlighting demand for brand-new remedies - Technology Life
Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - Town Celeb
Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - Luxuryfolks
Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - Washington Dailies
Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions – Adfero News
Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - RockingTimes
Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - Health Astros
Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions • CAPTIS
Pingback: AI Weekly: AI mannequin coaching prices on the rise, highlighting want for brand new options » Best IND News
Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - sfgtechnology.com
Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - thecowbotchannel.com
Pingback: AI Weekly: los costos de entrenamiento del modelo de AI aumentan, lo que destaca la necesidad de nuevas soluciones - Blog | Internet, Cibersegurdad, Digitalización y Tecnología
Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions – 365ViralBuzz
Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - The News Global
Pingback: AI Weekly: AI mannequin coaching prices on the rise, highlighting want for brand new options – Kabar kabari
Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - Cazy Boy Tech
Pingback: AI Weekly: AI mannequin coaching prices on the rise, highlighting want for brand spanking new options – Kilden Info
Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions – Omni Innotech
Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions | TechieTricks.com
Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - AI Caosuo
Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions – Eco Tech Digital
Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions – VentureBeat - AI Caosuo
Pingback: AI Weekly: AI model training costs on the rise, highlighting need for new solutions - epicapplications
Pingback: 4 Components of Modern Data Protection and Content Inspection
Pingback: Overcoming High AI Development Costs with a Hybrid Approach - Expert.ai - Get your daily dose of technews
Pingback: Mengatasi Biaya Pengembangan AI yang Tinggi dengan Pendekatan Hybrid - Professional.ai - hapidzfadli
Pingback: Expert ai S p A : How a Hybrid Approach Can Mitigate High Development Costs of NLP Models – Marketscreener.com - AI Caosuo