Welcome to mini-RAG Ep. 25 in the series "minirag: From Notebook to Production"! In this video, we will continue learning Celery and how integrate it into your Python backend – ideal for ML workflows, RAG pipelines, or any production-grade system that needs asynchronous task processing.
Whether you're building a Retrieval-Augmented Generation (RAG) app, an API service, or automating ML pipelines – Celery is the go-to tool for distributed task queues in Python.
== What You’ll Learn:
00:00 – Step 1 Recap
02:35 – Celery Flower Monitoring Dashboard
09:15 – Secure a Flower Dashboard
13:17 – Refactor Data Indexing Endpoint to a Celery task
25:04 – Celery Workflows and Chains
24:30 – Monitoring a Celery Chain via Flower
45:15 – What does an Idempotent task mean?
49:10 – Design an Idempotent tasks manager
53:00 – Idempotent payload signature
01:10:26 – Idempotent task manager within a Late-Akn Celery tasks
01:27:00 – Cleaning up with Celery Beat
01:41:00 – Docker services for Celery worker, beat, and flower
01:48:08 – Conclusion
Codes:
https://github.com/bakrianoo/mini-rag/tree/tut-017
== Technologies Covered:
Celery (Python Task Queue)
FastAPI
Docker & Docker Compose
Redis & RabbitMQ
Async Background Task Architecture
