Redfin Analytics|python ETL pipeline with airflow|Data Engineering Project|Snowpipe|Snowflake|Part 1

This is the part 1 of this Redfin Real Estate Data Analytics python ETL data engineering project using Apache Airflow, Snowpipe, snowflake and AWS services. In this Redfin Real Estate Data Analytics python ETL data engineering project, you will learn how to connect to the Redfin data center data source to extract real estate data using python after which we will transform the data using pandas and load it into an Amazon S3 bucket. The raw data will also be loaded into an Amazon S3 bucket. As soon as the transformed data lands inside the AWS S3 bucket, Snowpipe would be triggered which would automatically run a COPY command to load the transformed data into a snowflake data warehouse table. We would then connect PowerBi to the snowflake data warehouse to then visualize the data to obtain insight. Apache airflow would be used to orchestrate and automate this process. Apache Airflow is an open-source platform used for orchestrating and scheduling workflows of tasks and data pipelines. We would install the Apache-airflow on our EC2 instance to orchestrate the pipeline. Remember the best way to learn data engineering is by doing data engineering - Get your hands dirty! If you have any questions or comments, please leave them in the comment section below. Please don’t forget to LIKE, SHARE, COMMENT and SUBSCRIBE to our channel for more AWESOME videos. **Books I recommend** 1. Grit: The Power of Passion and Perseverance https://amzn.to/3EZKSgb 2. Think and Grow Rich!: The Original Version, Restored and Revised: https://amzn.to/3Q2K68s 3. The Book on Rental Property Investing: How to Create Wealth With Intelligent Buy and Hold Real Estate Investing: https://amzn.to/3LLpXRy 4. How to Invest in Real Estate: The Ultimate Beginner's Guide to Getting Started: https://amzn.to/48RbuOb 5. Introducing Python: Modern Computing in Simple Packages https://amzn.to/3Q4driR 6. Python for Data Analysis: Data Wrangling with pandas, NumPy, and Jupyter 3rd Edition: https://amzn.to/3rGF73G ***************** Commands used in this video ***************** Check out my github Repo https://github.com/YemiOla/data_engineering_project_redfin__dataanalytics ***************** USEFUL LINKS ***************** 1. Zillow Data Analytics (RapidAPI) | End-To-End Python ETL Pipeline | Data Engineering Project |Part 1 https://www.youtube.com/watch?v=j_skupZ3zw0 2. https://www.redfin.com/news/data-center/ 3. How to Build and Automate loading data from S3 to Snowflake with email notification using airflow https://www.youtube.com/watch?v=Trn8gg9IlRs 4. How to remotely SSH (connect) Visual Studio Code to AWS EC2 https://www.youtube.com/watch?v=sQQjMnEkGjs 5. Monitor workflow with slack alert upon DAG failure | Airflow Tutorial https://www.youtube.com/watch?v=jVqnKge0AJQ 6. How to send out email alert ON RETRY and ON FAILURE in Apache airflow | Airflow Tutorial https://www.youtube.com/watch?v=Its_66azEy0 7. How to build and automate a python ETL pipeline with airflow on AWS EC2 | Data Engineering Project https://www.youtube.com/watch?v=uhQ54Dgp6To 8. https://docs.snowflake.com/en/sql-reference/sql/create-file-format 9. https://docs.snowflake.com/en/sql-reference/sql/create-stage 10. https://docs.snowflake.com/en/sql-reference/sql/copy-into-table 11. https://docs.snowflake.com/en/sql-reference/sql/create-pipe 12. https://docs.snowflake.com/en/sql-reference/sql/desc-table 13. https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/operators/python/index.html 14. Customer Churn Data Analytics|Data Pipeline using Apache Airflow, Glue, S3, Redshift, PowerBI | Part 3 https://www.youtube.com/watch?v=HKVLqghypsA 15. How to build a pipeline to create table and insert records on snowflake with airflow on AWS EC2 https://www.youtube.com/watch?v=R8FTRFr2MpM 16. https://docs.snowflake.com/en/user-guide/data-load-snowpipe-intro 17. PostgreSQL Playlist: https://www.youtube.com/watch?v=oFaLUCWRnRE&list=PLACD_PaYcVF09khO58CISr08Uy6w3cAIF 18. Apache Airflow Playlist https://www.youtube.com/watch?v=uhQ54Dgp6To&list=PLACD_PaYcVF1Hzzc1Ds56bD7oUkfiL_Lv 19. Download PowerBI https://www.microsoft.com/en-US/download/details.aspx?id=58494 DISCLAIMER: This video and description have affiliate links. This means when you buy through one of these links, we will receive a small commission and this is at no cost to you. This will help support us to continue making awesome and valuable contents for you.