Realtime Change Data Capture Streaming | End to End Data Engineering Project

In this video, we dive deep into the world of Change Data Capture (CDC) and how it can be implemented for real-time data streaming using a powerful tech stack. You will use the integration of technologies like Docker, Postgres, Debezium, Kafka, Apache Spark, and Slack to create an efficient and responsive data pipeline. Master Data Engineering by enrolling at datamasterylab.com Timestamps: 0:00 Introduction 0:35 The system architecture 14:40 Getting live data into postgres db 31:00 Connecting to Postgres with Debezium and Kafka from the UI 34:35 Previewing Debezium data on Kafka 37:25 Getting full data from Postgres with Debezium 39:55 Setting up debezium connector from the terminal 41:00 Handling decimal values on debezium 46:00 Getting the user that changed data on postgres with time 53:17 Creating a more robust data capture on postgres 1:03:31 Outro Don't forget to SUBSCRIBE, LIKE, COMMENT and SHARE for more exciting videos! Resources: Full Code: https://github.com/airscholar/changecapture-e2e.git Medium Article: https://medium.com/@yusuf.ganiyu/change-data-capture-cdc-realtime-streaming-with-postgres-debezium-kafka-apache-spark-and-slack-42f6ee74bc1c Data Mastery: datamasterlab.com 🔹 Connect with Us: Follow us on Twitter: x.com/@DataMasterylab Follow us on LinkedIn: https://www.linkedin.com/in/yusuf-ganiyu-b90140107 Tags: Change Data Capture, CDC, Real-Time Streaming, Docker, Postgres, Debezium, Kafka, Apache Spark, Slack, Data Engineering, Data Pipeline, Tech Tutorial, Software Development, Data Streaming, IT Education, End to End Data Engineering Hashtags: #ChangeDataCapture #RealTimeData #Docker #Postgres #Debezium #Kafka #ApacheSpark #Slack #DataEngineering #TechTutorial #SoftwareDevelopment #DataStreaming #ITEducation #bigdataanalytics #EndToEndDataEngineeringProject