AI21 Labs Asks: How Much Does It Cost to Train NLP Models?AI21 Labs Co-CEO, Stanford University Professor of Computer Science (emeritus), and AI Index initiator Yoav Shoham compared three different-sized Google BERT language models on the 15 GB Wikipedia and Book corpora, evaluating both the cost of a single training run and a typical, fully-loaded model cost.
The team estimated fully-loaded cost to include hyperparameter tuning and multiple runs for each setting:
- $2.5k — $50k (110 million parameter model)
- $10k — $200k (340 million parameter model)
- $80k — $1.6m (1.5 billion parameter model)
Paper: https://arxiv.org/pdf/2004.08900.pdf
BlogPost: https://bit.ly/2SllsBK
#ai21 #nlp #gpu #bert #google
