#280 Native sparse attention from DeepSeek

Long-context modeling is crucial for next-generation language models, yet the high computational cost of standard attention mechanisms poses significant computational challenges. Sparse attention offers a promising direction for improving efficiency while maintaining model capabilities. This work presents NSA, a natively trainable sparse attention mechanism that integrates algorithmic innovations with hardware-aligned optimizations to achieve efficient long-context modeling. NSA employs a dynamic hierarchical sparse strategy, combining coarse-grained token compression with fine-grained token selection to preserve both global context awareness and local precision. The approach advances sparse attention design with two key innovations: (1) It achieves substantial speedups through arithmetic intensity-balanced algorithm design, with implementation optimizations for modern hardware. (2) It enables end-to-end training, reducing pretraining computation without sacrificing model performance. Experiments demonstrate that models pretrained with NSA maintain or exceed full attention models across general benchmarks, long-context tasks, and instruction-based reasoning. Meanwhile, NSA achieves substantial speedups over full attention on 64k-length sequences across decoding, forward propagation, and backward propagation, validating its efficiency throughout the model lifecycle. In this video, I talk about the following: How does Native Sparse Attention work? How do token compression, token selection and sliding window work in NSA? How does Native Sparse Attention perform? For more details, please look at https://arxiv.org/pdf/2502.11089 Yuan, Jingyang, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie et al. "Native sparse attention: Hardware-aligned and natively trainable sparse attention." arXiv preprint arXiv:2502.11089 (2025). Thanks for watching! LinkedIn: http://aka.ms/manishgupta HomePage: https://sites.google.com/view/manishg/