Data Wrangling  by Removing Duplication

Pandas duplicates() method helps us to remove duplicate values from Large Data. An important part of Data Wrangling is removing Duplicate values from the large data set.

Syntax: DataFrame.duplicated(subset=None, keep=’first’)

Here subset is the column value where we want to remove the Duplicate value.

In keeping, we have 3 options :

• if keep =’first’ then the first value is marked as the original rest of all values if occur will be removed as it is considered duplicate.
• if keep=’last’ then the last value is marked as the original rest the above same values will be removed as it is considered duplicate values.
• if keep =’false’ all the values which occur more than once will be removed as all are considered duplicate values.

For example, A University will organize the event. In order to participate Students have to fill in their details in the online form so that they will contact them. It may be possible that a student will fill out the form multiple times. It may cause difficulty for the event organizer if a single student will fill in multiple entries. The Data that the organizers will get can be Easily Wrangles by removing duplicate values.

Creating a Student Dataset who want to participate in the event:

Output:

Removing Duplicate data from the Dataset using Data wrangling:

Output:D

Creating New Datasets Using the Concatenation of Two Datasets In Data Wrangling. 

We can join two dataframe in several ways. For our example in Concanating Two datasets, we use pd.concat() function.  

Creating Two Dataframe For Concatenation.

We will join these two dataframe along axis 0.

output:

    Name    Age    Address    Qualification    Mobile No    Salary
0    Jai        27    Nagpur          Msc            97.0        NaN
1    Princi    24    Kanpur        MA            91.0        NaN
2    Gaurav    22    Allahabad    MCA            58.0        NaN
3    Anuj    32    Kannuaj        Phd            76.0        NaN
4    Gaurav    22    Allahabad    MCA            NaN            1000.0
5    Anuj    32    Kannuaj        Phd            NaN            2000.0
6    Dhiraj    12    Allahabad    Bcom        NaN            3000.0
7    Hitesh    52    Kannuaj        B.hons        NaN            4000.0

Note:- We can see that data1 does not have a salary column so all four rows of new dataframe res are Nan values.

Data Wrangling is the process of gathering, collecting, and transforming Raw data into another format for better understanding, decision-making, accessing, and analysis in less time. Data Wrangling is also known as Data Munging.