Cleaning DataLesson 13Free preview

Removing Duplicates

This lesson is part of Pandas · All levels

Summary

The duplicate line makes the data count double so that the statistics become biased. The pandas provide duplicated to detect the twin lines and drop_duplicates to discard them. You can also select a specific column as a unique reference if the entire row is not identical.

Key points

Duplicate is a line that appears more than once
Duplicated marks the twin lines with a boolean
Drop_duplicates delete duplicates with the keep option
Column subset can be set as a unique reference

Full lesson

Code examples and quizzes

Full lessons, code examples, and quizzes are available to subscribers. 5 questions.