Llama 3 training: 24K H100s and 15T tokens
Posted by Agastya Rishi · 2024-03-30
Meta reveals Llama 3 training infrastructure: 24,000 H100 GPUs, 15 trillion tokens, custom tokenizer with 128K vocabulary. The compute requirements are staggering but the model quality justifies it. https://github.com/meta-llama/llama3