Remove duplicate lines and keep the order

What counts as a duplicate?

Two lines are duplicates when they match character for character. That sounds simple, but real lists are full of near-matches: an email typed with capitals, a code copied with a space on the end, a name with an accent typed two different ways. Two switches decide how strict the match is.

• Match case. On means "Jo@Example.nz" and "jo@example.nz" are different lines. Off treats them as the same. For email addresses and most names, off is usually what you want; for codes where case matters, leave it on.
• Ignore spaces at the start and end. On means a line with a stray trailing space still matches its clean twin. The space itself is kept in whichever copy survives, unless you also trim.

Accented letters are compared in a standard form, so "é" typed as one character and "é" built from "e" plus an accent mark are treated as the same.

Worked example: a mailing list

Here are six addresses collected from two sign-up sheets:

jo@example.nz
sam@example.nz
Jo@Example.nz
lee@example.nz
sam@example.nz 
jo@example.nz

With "match case" on and spaces counted, only exact repeats go, leaving 5 lines. With "match case" off and surrounding spaces ignored, the capitalised and space-padded versions are recognised as repeats too, leaving 3:

jo@example.nz
sam@example.nz
lee@example.nz

Keep the first copy or the last?

Order is always preserved; the question is which copy stays. Keeping the first suits lists built over time, where the earliest entry is the original. Keeping the last suits logs and exports where later lines are more up to date. With the lines "v1 draft, final copy, v1 draft, notes, final copy", keeping the first gives v1 draft, final copy, notes, while keeping the last gives v1 draft, notes, final copy.

Mistakes that leave duplicates behind

• Invisible characters. Text copied from web pages can contain non-breaking spaces or tabs. Turn on trimming and collapsing (under "Tidy spaces") so they don't make a line look unique.
• Sorting first, by hand. You don't need to sort before removing duplicates; the cleaner finds repeats anywhere in the list. Sort afterwards if you want alphabetical order.
• Different spellings. "St Heliers" and "Saint Heliers" aren't duplicates to a computer. The cleaner removes exact matches only.

Big lists

Lists of hundreds of thousands of lines are fine; anything over 200,000 characters is processed in the background so the page doesn't freeze. The tool counts how many duplicate lines it removed. Your list stays in your browser and is never uploaded. For more cleaning options in one pass, open the full text cleaner; to put the unique lines in order, see sorting lines alphabetically.

Last reviewed: 2 October 2026