How to use DataFrame.select() In Python
Here we will use select() function, this function is used to select the columns from the dataframe
Syntax: dataframe.select(columns)
Where dataframe is the input dataframe and columns are the input columns
Example 1: Change a single column.
Let us convert the `course_df3` from the above schema structure, back to the original schema.
Python
from pyspark.sql.types import StringType, BooleanType, IntegerType course_df4 = course_df3.select( course_df3.Name, course_df3.Course_Name, course_df3.Duration_Months, (course_df3.Course_Fees.cast(IntegerType())) .alias( 'Course_Fees' ), (course_df3.Start_Date.cast(StringType())) .alias( 'Start_Date' ), (course_df3.Payment_Done.cast(BooleanType())) .alias( 'Payment_Done' ), ) course_df4.printSchema() |
Output:
root |-- Name: string (nullable = true) |-- Course_Name: string (nullable = true) |-- Duration_Months: long (nullable = true) |-- Course_Fees: integer (nullable = true) |-- Start_Date: string (nullable = true) |-- Payment_Done: boolean (nullable = true)
Example 2: Changing multiple columns to the same datatype.
Python
# Changing datatype of all the columns # to string type from pyspark.sql.types import StringType course_df5 = course_df.select( [course_df.cast(StringType()) .alias(c) for c in course_df.columns] ) course_df5.printSchema() |
Output:
root |-- Name: string (nullable = true) |-- Course_Name: string (nullable = true) |-- Duration_Months: string (nullable = true) |-- Course_Fees: string (nullable = true) |-- Start_Date: string (nullable = true) |-- Payment_Done: string (nullable = true)
Example 3: Changing multiple columns to the different datatypes.
Let us use the `course_df5` which has all the column type as `string`. We will change the column types to a respective format.
Python
from pyspark.sql.types import ( StringType, BooleanType, IntegerType, FloatType, DateType ) coltype_map = { "Name" : StringType(), "Course_Name" : StringType(), "Duration_Months" : IntegerType(), "Course_Fees" : FloatType(), "Start_Date" : DateType(), "Payment_Done" : BooleanType(), } # course_df6 has all the column # types as string course_df6 = course_df5.select( [course_df5.cast(coltype_map) .alias(c) for c in course_df5.columns] ) course_df6.printSchema() |
Output:
root |-- Name: string (nullable = true) |-- Course_Name: string (nullable = true) |-- Duration_Months: integer (nullable = true) |-- Course_Fees: float (nullable = true) |-- Start_Date: date (nullable = true) |-- Payment_Done: boolean (nullable = true)
How to Change Column Type in PySpark Dataframe ?
In this article, we are going to see how to change the column type of pyspark dataframe.
Creating dataframe for demonstration:
Python
# Create a spark session from pyspark.sql import SparkSession spark = SparkSession.builder.appName( 'SparkExamples' ).getOrCreate() # Create a spark dataframe columns = [ "Name" , "Course_Name" , "Duration_Months" , "Course_Fees" , "Start_Date" , "Payment_Done" ] data = [ ( "Amit Pathak" , "Python" , 3 , 10000 , "02-07-2021" , True ), ( "Shikhar Mishra" , "Soft skills" , 2 , 8000 , "07-10-2021" , False ), ( "Shivani Suvarna" , "Accounting" , 6 , 15000 , "20-08-2021" , True ), ( "Pooja Jain" , "Data Science" , 12 , 60000 , "02-12-2021" , False ), ] course_df = spark.createDataFrame(data).toDF( * columns) # View the dataframe course_df.show() |
Output:
Let’s see the schema of dataframe:
Python
# View the column datatypes course_df.printSchema() |
Output: