How to Remove Duplicate Lines from a List

Mailing lists, keyword exports and log files all collect duplicates. The Duplicate Line Remover keeps the first copy of each line and drops the rest.

What counts as a duplicate

That depends on three options:

• Case-insensitive treats “Red” and “red” as the same line. It is off by default.
• Trim before comparing ignores spaces at the start and end of a line, so “blue” and “blue ” match. On by default.
• Ignore empty lines leaves blank lines alone instead of treating them as repeats. On by default.

A worked example

Take six lines: red, blue, Red, blue, an empty line, green.

• With the defaults, only the second “blue” is removed. The report says “Removed 1 duplicate from 6 lines (5 remain)”, and “Red” stays because it differs in case.
• With case-insensitive on, “Red” is also removed: “Removed 2 duplicates from 6 lines (4 remain)”.

The order of the first occurrences is preserved, so the result is not sorted. If you also want an alphabetical list, run it through the line sorter afterwards.

Using the tool

1. Paste your list, one item per line.
2. Choose the options.
3. Read the report, then copy or download the cleaned list.

Things to watch

• Lines that look the same but contain a hidden difference, such as a non-breaking space, are treated as different. Clean those first with the extra spaces tool.
• Only whole lines are compared. A repeated word inside a line is not touched.

Frequently asked questions