scd2 in spark | Lec-24

In this video I have talked about slowly changing dimension type 2. Directly connect with me on:- https://topmate.io/manish_kumar25 Discord channel:- https://discord.gg/6r9QF6TE3F SCD Data:- customer_dim_data = [ (1,'manish','arwal','india','N','2022-09-15','2022-09-25'), (2,'vikash','patna','india','Y','2023-08-12',None), (3,'nikita','delhi','india','Y','2023-09-10',None), (4,'rakesh','jaipur','india','Y','2023-06-10',None), (5,'ayush','NY','USA','Y','2023-06-10',None), (1,'manish','gurgaon','india','Y','2022-09-25',None), ] customer_schema= ['id','name','city','country','active','effective_start_date','effective_end_date'] customer_dim_df = spark.createDataFrame(data= customer_dim_data,schema=customer_schema) sales_data = [ (1,1,'manish','2023-01-16','gurgaon','india',380), (77,1,'manish','2023-03-11','bangalore','india',300), (12,3,'nikita','2023-09-20','delhi','india',127), (54,4,'rakesh','2023-08-10','jaipur','india',321), (65,5,'ayush','2023-09-07','mosco','russia',765), (89,6,'rajat','2023-08-10','jaipur','india',321) ] sales_schema = ['sales_id', 'customer_id','customer_name', 'sales_date', 'food_delivery_address','food_delivery_country', 'food_cost'] sales_df = spark.createDataFrame(data=sales_data,schema=sales_schema) https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.persist.html https://spark.apache.org/docs/latest/rdd-programming-guide.html For more queries reach out to me on my below social media handle. Follow me on LinkedIn:- https://www.linkedin.com/in/manish-kumar-373b86176/ Follow Me On Instagram:- https://www.instagram.com/competitive_gyan1/ Follow me on Facebook:- https://www.facebook.com/MANISH12340 My Second Channel -- https://www.youtube.com/channel/UCqX5o-tLG33L3RaBIdOFBWA Interview series Playlist:- https://www.youtube.com/playlist?list=PLTsNSGeIpGnFBjVePu_6ZQVvmgHChBh5L My Gear:- Rode Mic:-- https://amzn.to/3RekC7a Boya M1 Mic-- https://amzn.to/3uW0nnn Wireless Mic:-- https://amzn.to/3TqLRhE Tripod1 -- https://amzn.to/4avjyF4 Tripod2:-- https://amzn.to/46Y3QPu camera1:-- https://amzn.to/3GIQlsE camera2:-- https://amzn.to/46X190P Pentab (Medium size):-- https://amzn.to/3RgMszQ (Recommended) Pentab (Small size):-- https://amzn.to/3RpmIS0 Mobile:-- https://amzn.to/47Y8oa4 ( Aapko ye bilkul nahi lena hai) Laptop -- https://amzn.to/3Ns5Okj Mouse+keyboard combo -- https://amzn.to/3Ro6GYl 21 inch Monitor-- https://amzn.to/3TvCE7E 27 inch Monitor-- https://amzn.to/47QzXlA iPad Pencil:-- https://amzn.to/4aiJxiG iPad 9th Generation:-- https://amzn.to/470I11X Boom Arm/Swing Arm:-- https://amzn.to/48eH2we My PC Components:- intel i7 Processor:-- https://amzn.to/47Svdfe G.Skill RAM:-- https://amzn.to/47VFffI Samsung SSD:-- https://amzn.to/3uVSE8W WD blue HDD:-- https://amzn.to/47Y91QY RTX 3060Ti Graphic card:- https://amzn.to/3tdLDjn Gigabyte Motherboard:-- https://amzn.to/3RFUTGl O11 Dynamic Cabinet:-- https://amzn.to/4avkgSK Liquid cooler:-- https://amzn.to/472S8mS Antec Prizm FAN:-- https://amzn.to/48ey4Pj