In this talk we’ll embark on a journey into the realm of large language models (LLMs), as we unravel the complexities of training 100-billion-plus parameter models on an unprecedented scale. We’ll delve into the intricacies of orchestrating thousands of GPUs to process trillions of tokens, discuss cutting-edge model architectures, training optimizations, hardware capabilities, and sophisticated orchestration and fault-tolerance strategies required to sustain weeks-long training runs.

This talk is suitable for ML researchers, engineers, and anyone curious about the “sausage making” behind large scale LLM training.