Conquering the Queue: Lessons from processing one billion Celery tasks

🔊 Recorded at PyCon DE & PyData 2025, April 24, 2025 https://2025.pycon.de/program/J8FLDN/ 🎓 A deep dive into scaling distributed task queues to process billions of operations, drawing from real-world implementation challenges and solutions. Speakers: Daniel Hepper Description: In this technical presentation, Daniel Hepper, CTO at Userlike, examines the challenges and solutions for processing large-scale task queues using Celery in Python web applications. Drawing from experience managing over 6 million daily Celery tasks, Hepper discusses core concepts of distributed task queues, broker selection, and failure handling strategies. The talk covers critical aspects of queue performance optimization, including intelligent queue design, worker configuration, and task prefetching. Hepper addresses common pitfalls such as stale state handling, transaction management, and deployment considerations in multi-tenant environments. Through practical examples from Userlike's messaging platform architecture, he demonstrates approaches to achieving system resiliency through proper monitoring, horizontal scaling, and graceful degradation. The presentation includes detailed analysis of task execution patterns and their impact on latency and throughput, particularly in mixed workload scenarios. Special attention is given to architectural decisions around broker selection, worker process configuration, and queue separation strategies. The discussion concludes with an overview of alternative task queue implementations and emerging solutions in the Django ecosystem, providing a comprehensive perspective on background task processing in modern web applications. ⭐️ About PyCon DE & PyData: The PyCon DE & PyData conference unite the Python, AI, and data science communities, offering a unique platform for collaboration and innovation. The PyCon DE & PyData 2025 conference, provided an exceptional experience, fostering deeper connections within the Python community while showcasing advancements in AI and data science. Attendees enjoyed a diverse and engaging program, solidifying the event as a highlight for Python and AI enthusiasts nationwide. Follow us: • LinkedIn: https://www.linkedin.com/company/28908640/ • X: https://www.x.com/pyconde Links: • Conference website: http://pycon.de • Other sessions: https://2025.pycon.de/talks/ The conference is organized by • Python Softwareverband e.V.: http://pysv.org • NumFOCUS Inc.: http://numfocus.org • Pioneers Hub gemeinnützige GmbH: http://pioneershub.org If you enjoyed this session, please like, comment, and subscribe to our channel for more insightful talks and discussions. Share this video with your network to spread the knowledge! Hashtags: #Python #PyConDE #PyData #OpenSource #AI #DataScience #MachineLearning #SoftwareDevelopment #LLMs #Community Acknowledgements: Special thanks to all the volunteers and sponsors who made this event possible. About: Python Softwareverband e.V.: PySV is a non-profit that promotes the use and development of Python in Germany through events, education, and advocacy, fostering an open Python community. NumFOCUS Inc. supports open-source scientific computing by providing financial and logistical support to key projects like NumPy and Jupyter, promoting sustainable development and collaboration. Pioneers Hub gemeinnützige GmbH: is a non-profit fostering innovation in AI and tech by connecting experts and promoting knowledge exchange through events and collaborative initiatives.