UnionAll() in PySpark
UnionAll() function does the same task as union() function but this function is deprecated since Spark “2.0.0” version. Hence, union() function is recommended.
Syntax:
dataFrame1.unionAll(dataFrame2)
Here,
- dataFrame1 and dataFrame2 are the dataframes
Example 1:
In this example, we have combined two data frames, data_frame1 and data_frame2. Note that the schema of both the data frames is the same.
Python3
# Python program to illustrate the # working of unionAll() function import pyspark from pyspark.sql import SparkSession spark = SparkSession.builder.appName('w3wiki.com').getOrCreate() # Creating a dataframe data_frame1 = spark.createDataFrame( [("Bhuwanesh", 82.98), ("Harshit", 80.31)], ["Student Name", "Overall Percentage"] ) # Creating another dataframe data_frame2 = spark.createDataFrame( [("Naveen", 91.123), ("Piyush", 90.51)], ["Student Name", "Overall Percentage"] ) # Union both the dataframes using unionAll() function answer = data_frame1.unionAll(data_frame2) # Print the union of both the dataframes answer.show() |
Output:
Example 2:
In this example, we have combined two data frames, data_frame1 and data_frame2. Note that the schema of both the data frames is different. Hence, the output is not the desired one as unionAll() function is ideal for datasets having the same structure or schema.
Python3
# Python program to illustrate the # working of union() function import pyspark from pyspark.sql import SparkSession spark = SparkSession.builder.appName('w3wiki.com').getOrCreate() # Creating a data frame data_frame1 = spark.createDataFrame( [("Bhuwanesh", 82.98), ("Harshit", 80.31)], ["Student Name", "Overall Percentage"] ) # Creating another data frame data_frame2 = spark.createDataFrame( [(91.123, "Naveen"), (90.51, "Piyush"), (87.67, "Hitesh")], ["Overall Percentage", "Student Name"] ) # Union both the dataframes using unionAll() function answer = data_frame1.unionAll(data_frame2) # Print the union of both the dataframes answer.show() |
Output:
Python PySpark – Union and UnionAll
In this article, we will discuss Union and UnionAll in PySpark in Python.