Remove Duplicate Lines
Paste a list and every repeated line after its first appearance is removed, while the remaining lines keep their original order. Matching can ignore letter case and stray spaces, and it all runs in your browser.
0 characters · 0 words · 0 lines
0 characters · 0 words · 0 lines
Runs entirely in this browser tab. Your text is never uploaded or stored.
What it does
Remove Duplicate Lines keeps the first occurrence of each line and drops later repeats while leaving the original order alone. Matching can ignore letter case and surrounding spaces, and blank lines can be kept or removed.
red blue Red green
red blue green
Who dedupes lists and why
Duplicate lines appear whenever lists are merged. A marketer combining two exported email lists, a researcher collecting keywords from several tools, a developer gathering URLs from log files, a teacher pasting the names from three sign-up forms: each ends up with entries that appear twice or more. Spreadsheets can remove duplicates too, but that means opening a file, selecting a column and finding the right menu; pasting into a text box is faster when the data is already on the clipboard.
Keeping the original order matters more than it seems. A list of steps, a ranking, or URLs in the order they were crawled loses meaning when it is sorted. Here, the first occurrence stays exactly where it was and later repeats are simply dropped, and a counter shows how many lines were removed so you can see whether the result is plausible.
How matching works, and the invisible differences
Each line is reduced to a comparison key and stored in a set as the list is read from top to bottom; a line whose key is already in the set is a duplicate. This takes one pass, so lists with hundreds of thousands of lines are handled quickly. It also differs from the classic Unix uniq command, which only removes duplicates that sit on adjacent lines and is therefore usually run after sort, losing the original order. The awk idiom awk '!seen[$0]++' does what this tool does.
Two lines can look identical and still differ. A trailing space, a tab instead of a space, or a non-breaking space copied from a web page makes the computer treat them as different strings. Turn on trimming to ignore spaces at both ends of each line, including non-breaking spaces. Turn off case sensitivity to treat "Paris" and "paris" as the same entry; the first spelling seen is the one kept. Blank lines can be kept, so paragraphs stay separated, or removed altogether.
After the cleanup
A deduplicated list often needs an order next: Sort Lines can arrange it alphabetically, in natural order or by length. If the list came from a PDF or an email with broken lines, fix those first with the other text tools, because a wrapped entry split over two lines cannot match its unbroken twin.
Frequently asked questions
- Which copy of a duplicate line is kept?
- The first one. Every later line with the same content is removed, and the remaining lines stay in their original order, so nothing is reshuffled.
- Why are two lines that look identical not removed?
- They differ in an invisible way, most often a trailing space, a tab or a non-breaking space. Turn on the trim option, or ignore case if only capitalization differs.
- How is this different from the uniq command?
- uniq only drops a line when it repeats the line directly above it, so it is normally paired with sort. This tool finds repeats anywhere in the text and keeps the original order.
- Can I remove duplicates and sort the list at the same time?
- Remove the duplicates here, then paste the result into Sort Lines. Doing it in two steps lets you keep the first-seen order when you need it and sort only when you want to.