Get S3 Data Process using Pyspark in Pycharm

To accelerate your career growth please join https://t.me/SparkTraining If you want to get a job opportunity in pySpark call: +91-8500002025 or https://wa.me/918500002025 or fill this form https://forms.gle/mJXHn9EieL1dAttq6 In this video I am explaining how to get data from S3, process data using Pyspark in Pycharm explaining in this video. You must have AWS knowledge to do it hands-on. from pyspark.sql import * ACCESS_KEY = """access keys must be in triple quotes""" SECRET_KEY = """otehrwise in that key if u have slash / its consider as escape char""" spark = (SparkSession.builder .appName("pyspark") .master("local[*]") .config("spark.hadoop.fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem") .config("spark.hadoop.fs.s3a.aws.credentia ls.provider", "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider") .config("spark.hadoop.fs.s3a.access.key", ACCESS_KEY) .config("spark.hadoop.fs.s3a.secret.key", SECRET_KEY) .config("spark.hadoop.fs.s3a.endpoint", "s3.ap-south-1.amazonaws.com") .config("spark.jars.packages", "org.apache.hadoop:hadoop-aws:3.3.4,com.amazonaws:aws-java-sdk-bundle:1.12.262") .getOrCreate()) data=r"s3a://bucket/folder/bank-full.csv" df=spark.read.format("csv").option("sep",";").option("header","true").load(data) df.show() //////////////////its old code//////// https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-dynamodb/1.12.183 https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-s3/1.12.183 https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-core/1.12.183 https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk/1.12.183 D:\bigdata\hadoop-3.2.2\share\hadoop\tools\lib\hadoop-aws-3.2.2.jar code ..,......... from pyspark.sql import * from pyspark.sql.functions import * spark = SparkSession.builder.master("local").appName("test").getOrCreate() Access_key_ID="KKIA2FDNHA" Secret_access_key="HhymrUkLCwWpu0SqO3/FDwwmw/0eB" # Enable hadoop s3a settings spark.sparkContext._jsc.hadoopConfiguration().set("com.amazonaws.services.s3.enableV4", "true") spark.sparkContext._jsc.hadoopConfiguration().set("fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem") spark.sparkContext._jsc.hadoopConfiguration().set("fs.s3a.aws.credentials.provider", \ "com.amazonaws.auth.InstanceProfileCredentialsProvider,com.amazonaws.auth.DefaultAWSCredentialsProviderChain") spark.sparkContext._jsc.hadoopConfiguration().set("fs.AbstractFileSystem.s3a.impl", "org.apache.hadoop.fs.s3a.S3A") spark.sparkContext._jsc.hadoopConfiguration().set("fs.s3a.access.key",Access_key_ID) spark.sparkContext._jsc.hadoopConfiguration().set("fs.s3a.secret.key",Secret_access_key) spark.sparkContext._jsc.hadoopConfiguration().set("fs.s3a.endpoint", "s3.ap-south-1.amazonaws.com") data="s3a://s3databucket/input/us-500.csv" df=spark.read.format('csv').option("header","true").option("inferSchema","true").load(data) df.show()