#293 LLaDA: Large Language Diffusion Models with Masking

The capabilities of large language models (LLMs) are widely regarded as relying on autoregressive models (ARMs). This work challenges that notion by introducing LLaDA, a diffusion model trained from scratch under the pre-training and supervised finetuning (SFT) paradigm. LLaDA employs a forward data masking process and a reverse generation process, parameterized by a Transformer to predict masked tokens. It provides a principled generative approach for probabilistic inference by optimizing a likelihood lower bound. Across extensive benchmarks on general tasks, math, code, and more, LLaDA demonstrates strong scalability and performs comparably to self-constructed ARM baselines. Remarkably, LLaDA 8B is competitive with strong LLMs like LLaMA3 8B in in-context learning and, after SFT, exhibits impressive instruction-following abilities in case studies such as multiturn dialogue. Moreover, LLaDA addresses the reversal curse, surpassing GPT-4o in a reversal poem completion task. These findings show the promise of diffusion models for language modeling at scale and challenge the common assumption that core LLM capabilities discussed above inherently depend on ARMs. In this video, I talk about the following: What is LLaDA? How does LLaDA work? How is LLaDA trained and how does it perform? For more details, please look at https://arxiv.org/pdf/2502.09992 and https://ml-gsai.github.io/LLaDA-demo/ and https://llada.pro/ and https://huggingface.co/spaces/multimodalart/LLaDA Nie, Shen, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. "Large language diffusion models." arXiv preprint arXiv:2502.09992 (2025). Thanks for watching! LinkedIn: http://aka.ms/manishgupta HomePage: https://sites.google.com/view/manishg/