After I post my last article โ๐๐ผ ๐๐ผ๐ ๐ธ๐ป๐ผ๐ ๐ต๐ผ๐ ๐บ๐๐ฐ๐ต ๐ถ๐ ๐ฐ๐ผ๐๐๐ ๐๐ผ ๐๐ฟ๐ฎ๐ถ๐ป ๐๐ฎ๐ฟ๐ด๐ฒ ๐๐ฎ๐ป๐ด๐๐ฎ๐ด๐ฒ ๐ ๐ผ๐ฑ๐ฒ๐น๐ (๐๐๐ ๐)?โ, I got lot of replies & queries that โwhy & where these costs are goingโ. So, with this article Iโm trying to explain this. I hope everyone will get answer of their queries.
We estimated a cost breakdown to develop key frontier models such as GPT-4 and Gemini Ultra, including R&D staff costs and compute for experiments. We found that most of the development cost is for the hardware at 47โ67%, but R&D staff costs are substantial at 29โ49%, with the remaining 2โ6% going to energy consumption.
If the trend of growing training costs continues, the largest training runs will cost more than a billion dollars by 2027
Hardware costs include AI accelerator chips (GPUs or TPUs), servers, and interconnection hardware
We also estimate the energy consumption of the hardware during the final training run of each model.
Using this method, we estimated the training costs for 45 frontier models (models that were in the top 10 in terms of training compute when they were released) and found that the overall growth rate is 2.4x per year.