Llama 3 training: 24K H100s and 15T tokens

Posted by Agastya Rishi · 2024-03-30

Meta reveals Llama 3 training infrastructure: 24,000 H100 GPUs, 15 trillion tokens, custom tokenizer with 128K vocabulary. The compute requirements are staggering but the model quality justifies it. https://github.com/meta-llama/llama3

Explore GitVibes