How to Remove Duplicate Lines From a List

Duplicate lines usually arrive when two lists get pasted together: last month's sign-up sheet plus this month's, a keyword export from three different searches, a crawl of a site that links to the same page from every footer. Finding the repeats by eye stops being practical somewhere around a hundred lines. The tool on this page does it in one pass, and the rest of this guide explains exactly what it treats as a duplicate, because that is where most surprises come from.

What counts as a duplicate here

Every tool that removes duplicates has to decide when two lines are "the same." Spreadsheets, text editors and command-line tools all decide slightly differently. This one follows five rules:

1. Lines are split on line breaks only. A comma, tab or semicolon is not a separator, so red, blue is one line.
2. Spaces and tabs at the start and end of each line are trimmed before comparing. apple and apple   match. Spaces inside a line still count, so new york and new  york (two spaces) do not.
3. Comparison is exact, character for character, unless "Ignore capitalization" is ticked. With it ticked, the first spelling the tool meets is the one it keeps.
4. The first occurrence wins and keeps its position. Order is only changed if you pick one of the sort buttons.
5. Blank lines are dropped and counted separately, so the summary line tells you how many real duplicates were removed.

Rule 3 is the one people trip over most. Excel's Remove Duplicates command and Google Sheets' Data cleanup both ignore letter case, and Google's help page says so directly: cells that differ only in capitalization are treated as duplicates. That is the right default for names and email addresses, and the wrong one for things like file paths, passwords or code identifiers, where Config.json and config.json really are different. Here you choose.

A worked example

Say you merged two sign-up lists and ended up with these eight lines (the fourth has two trailing spaces, and line five is blank):

maria.lopez@example.com
j.chen@example.com
Maria.Lopez@example.com
sam.okafor@example.com
(blank)
j.chen@example.com
sam.okafor@example.com
priya@example.com

With "Ignore capitalization" off, Remove duplicates returns five lines and reports 2 duplicates removed and 1 blank line dropped. Maria.Lopez@example.com survives because its capitals make it a different string. With the box ticked, you get four lines:

maria.lopez@example.com
j.chen@example.com
sam.okafor@example.com
priya@example.com

The trailing spaces on Sam's first entry did not stop it matching the second one, because both were trimmed before comparing. Email addresses are case-insensitive in practice, so for a mailing list the ticked result is the one you want.

The four buttons

Remove duplicates keeps your original order. Use it when order carries meaning: a ranked list, a playlist, steps in a procedure, rows you will paste back next to other columns.

Dedupe + sort A–Z and Z–A sort with your browser's language rules rather than raw character codes. In practice that means apple, Apple and banana sort the way a person would expect, instead of every capitalized word jumping ahead of every lowercase one, which is what a plain character-code sort does.

Dedupe + sort by length puts the shortest lines first. Lines of equal length keep the order they had. It is handy for spotting junk entries, since stray fragments like x or - float to the top, and for keyword lists, where the short entries tend to be the broad terms. If you only want to reorder without removing anything, the List Sorter has more sort modes and leaves duplicates alone.

When two lines look the same but both stay

If the output still shows what looks like a repeat, one copy almost always contains a character that looks like another one. The usual suspects, especially in text copied from web pages, PDFs or word processors:

• A non-breaking space in the middle of the line instead of a normal space. It is invisible and prints the same, but it is a different character.
• A curly apostrophe or quote (’ or “) in one line and a straight one (' or ") in the other.
• A zero-width space, which takes up no room at all on screen. These are common in text copied from chat apps and some websites.
• Different dashes: a hyphen, an en dash and a minus sign look nearly identical in many fonts.

The quickest fix is to paste the text into a plain-text editor first, or run it through the whitespace cleaner to normalize spacing, then deduplicate again.

Duplicates that are not exact copies

Some lists contain entries that mean the same thing without being the same text, and no line-by-line tool will catch them. URLs are the classic case. https://example.com/page, https://example.com/page/ and https://example.com/page?utm_source=news usually load the same page, but they are three different lines. Strip tracking parameters and settle on one trailing-slash style with find and replace before you deduplicate.

Log files have the opposite problem: every line starts with a timestamp, so nothing is ever a duplicate. Remove the timestamp column first (a spreadsheet's split-text feature does this well), then deduplicate what remains.

Other ways to remove duplicate lines

If the data already lives somewhere else, it may be quicker to deduplicate it there. Note that most of these do not trim spaces first, so results can differ slightly from this tool.

• Excel: select the column, then Data > Data Tools > Remove Duplicates. Microsoft's support article suggests filtering for unique values first so you can check the result before anything is deleted. Excel 365 and 2021 also have a =UNIQUE() formula that leaves the original column untouched.
• Google Sheets: Data > Data cleanup > Remove duplicates, or =UNIQUE(A:A) in an empty column.
• Notepad++: Edit > Line Operations > Remove Duplicate Lines. The neighbouring "Remove Consecutive Duplicate Lines" only catches repeats that sit directly next to each other.
• Mac or Linux terminal: sort -u list.txt gives a sorted, unique list. To keep the original order, use awk '!seen[$0]++' list.txt. Running uniq on its own only removes adjacent repeats, which is why it is normally paired with sort.
• Python: list(dict.fromkeys(lines)) removes duplicates and keeps order. list(set(lines)) also removes them but scrambles the order.

Size limits and privacy

The tool is a short piece of JavaScript that runs in your browser tab. Nothing is sent to a server, so there is no upload limit and no copy of your list stored anywhere. Speed depends on your device. In testing, the same code processed a 200,000-line list with 50,000 unique entries in under a tenth of a second. You can confirm that nothing leaves your machine by opening your browser's developer tools, switching to the Network tab and pressing the button. No request appears.

Frequently asked questions

Is the comparison case-sensitive?

Yes, by default. "Apple" and "apple" are kept as two lines. Tick "Ignore capitalization" to treat them as one; the tool keeps whichever spelling appeared first.

Does it remove blank lines?

Yes. Empty lines and lines containing only spaces are dropped. They are counted separately in the summary, so the duplicate count only includes real repeats.

Why are two identical-looking lines both still there?

One of them almost certainly contains a look-alike character: a non-breaking space, a curly apostrophe, a zero-width space or a different kind of dash. Paste the text through a plain-text editor or the whitespace cleaner and try again.

Can I deduplicate a comma-separated list?

Put each item on its own line first. In most editors, find-and-replace can swap , for a line break; in a spreadsheet, split the text into columns and then transpose it into one column.

Is my data private?

Yes. The list is processed in your browser and never uploaded, so it is safe to use with customer emails or internal data.