Data Engineering Process Fundamentals - Building a Cloud Based Data Pipeline

For this technical session, we look at a data engineering process and the related cloud technologies that are needed to build a data pipeline from CSV files to data visualization for Big Data scenarios for real use cases. - Follow this GitHub repo during the presentation: (Give it a star) https://github.com/ozkary/data-engineering-mta-turnstile - Read more information on my blog at: https://www.ozkary.com/2023/03/data-engineering-process-fundamentals.html - Chapters: 0:00:00 Data Engineering Process Overview 0:05:42 Discovery Process 0:11:39 Design and Planning 0:20:19 Data Pipeline and Orchestration 0:33:31 Data Warehouse & Transformation 0:51:10 Data analysis & visualization 1:01:54 Closing thoughts Some of the technologies that we will be covering: - Data Lakes - Data Warehouse - Data Analysis and Visualization - Python - Jupyter Notebook - SQL - More