Connect Apache Airflow to Snowflake Data Warehouse

This video explains how easily we can create Airflow-Snowflake connection and execute SQL commands in your Snowflake data warehouse. Commands & Python code: -------------------------------------- sudo apt update sudo apt install python3-pip sudo apt install sqlite3 sudo apt-get install libpq-dev pip3 install --upgrade awscli sudo pip3 install virtualenv virtualenv venv source venv/bin/activate pip install "apache-airflow[postgres]==2.5.0" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.5.0/constraints-3.7.txt" pip install pandas apache-airflow-providers-snowflake==2.1.0 snowflake-connector-python==2.5.1 snowflake-sqlalchemy==1.2.5 airflow db init sudo apt-get install postgresql postgresql-contrib sudo -i -u postgres psql CREATE DATABASE airflow; CREATE USER airflow WITH PASSWORD 'airflow'; GRANT ALL PRIVILEGES ON DATABASE airflow TO airflow; exit exit ls cd airflow sed -i 's#sqlite:////home/ubuntu/airflow/airflow.db#postgresql+psycopg2://airflow:airflow@localhost/airflow#g' airflow.cfg sed -i 's#SequentialExecutor#LocalExecutor#g' airflow.cfg airflow db init airflow users create -u airflow -f airflow -l airflow -r Admin -e [email protected] User id --airflow password--admin@123! mkdir /home/ubuntu/dags cd airflow vi airflow.cfg change the below properties -- dags_folder = /home/ubuntu/dags load_examples = False In Snowflake execute the below codes: ---------------------------------------- DROP DATABASE IF EXISTS RAMU; create DATABASE RAMU; USE DATABASE RAMU; SELECT * FROM source_table; Now put the python code(check pinned comment) in dags directory (/home/ubuntu/dags) Execute the below code to start the server: ------------------------------------------------ airflow db init airflow webserver in a new terminal source venv/bin/activate airflow scheduler Create the Snowflake Connection in Airflow & execute the dag. Check this playlist for more Data Engineering related videos: https://youtube.com/playlist?list=PLjfRmoYoxpNopPjdACgS5XTfdjyBcuGku Apache Kafka form scratch https://youtube.com/playlist?list=PLjfRmoYoxpNrs0VmIq6mOTqXP52RfZdRf Snowflake Complete Course from scratch with End-to-End Project with in-depth explanation-- https://doc.clickup.com/37466271/d/h/13qc4z-104/d4346819bd8d510 🙏🙏🙏🙏🙏🙏🙏🙏 YOU JUST NEED TO DO 3 THINGS to support my channel LIKE SHARE & SUBSCRIBE TO MY YOUTUBE CHANNEL