Text Tools

Remove duplicate lines.

Paste a list and get it back with repeated lines removed. Choose whether case and spacing matter, keep the first or last copy, sort the result, or show only the lines that repeat.

Advertisement

Input

Result

Paste a list to remove its duplicates.

Advertisement

About this duplicate line remover

Each line is compared with every other line, and repeated lines are removed so every value appears once. It works on emails, URLs, keywords, SKUs, log lines and any other list that has one item per line.

Worked example

apple
banana
Apple
banana
cherry

With the defaults the output is apple, banana, Apple, cherry (one per line): "Apple" and "apple" are different until you tick Ignore case, and then the first spelling wins. Switching Show to "Only lines that repeat" returns just banana.

Options

Limits

Matching is exact after the options above; similar but not identical lines (a typo, different punctuation) are kept. Input is limited to 5 MB. Processing runs in your browser.

Why duplicates appear, and which option fixes which kind

Duplicate lines creep in when lists are merged, exported twice or copied from several sources: email addresses collected from three campaigns, keyword lists exported from different tools, URLs pulled from several sitemaps, or log lines repeated by a retry loop. Not every duplicate looks identical, so the options matter more than the button.

Worked example with numbers

Take a ten-line keyword list: seo tools, SEO Tools, seo tools (trailing space), keyword research, keyword research, backlinks, an empty line, backlinks, rank tracker, seo tools. With the defaults (trim on, ignore case off, empty lines removed) you get five lines: seo tools, SEO Tools, keyword research, backlinks and rank tracker. The trailing-space copy and the second plain seo tools merge with the first, but SEO Tools survives because case matters. Tick Ignore case and the result drops to four lines. Switch Show to "Only lines that repeat" and you get the three values that occurred more than once: seo tools, keyword research and backlinks. "Only lines that never repeat" returns SEO Tools and rank tracker, the two values that occur once.

Keep first or keep last?

Keeping the first occurrence preserves the order in which each value first appeared, which is what most people want. Keeping the last occurrence matters when later lines are corrections or newer entries, for example a log where the latest status for an ID is the one you care about. When Ignore case is on, this choice also decides which spelling survives: keep first returns Apple from Apple, apple, keep last returns apple.

Sorting and what "natural" order means

With Sort set to A to Z, items are put in natural order, so item2 comes before item10 instead of after it, as a plain character-by-character sort would put it. Z to A reverses that, By length puts the shortest lines first and Shuffle randomises the order. If you only need a sorted, deduplicated list on a Unix command line, sort -u does both: POSIX defines its -u option as suppressing all but one in each set of lines having equal keys, and -f folds lower case to upper case so case is ignored.

How this compares with the uniq command

The POSIX uniq utility compares adjacent lines only: the specification says repeated lines are not detected if they are not adjacent, so people sort first and pipe into uniq. uniq -d prints only repeated lines and uniq -u only lines that are not repeated, and uniq -c prefixes each line with a count. This page does not require sorted input and keeps the original order by default, and the "Only lines that repeat" and "Only lines that never repeat" choices correspond to the -d and -u behaviours. It does not show per-line counts; if you need frequencies, use sort | uniq -c in a terminal.

Common mistakes

Limits

Input is limited to 5 MB. Matching is exact once the chosen options are applied, so the tool will not detect typos or fuzzy matches. It compares whole lines, not words inside them. To see what changed between two versions of a list, use the text diff. Everything runs in your browser.

Frequently asked questions

How do I remove duplicate lines but keep the original order?

Leave Sort on "Keep original order". The first occurrence of each line stays where it was and later copies are removed.

How do I find which lines are duplicated?

Set Show to "Only lines that repeat". The status line also counts how many distinct lines appeared more than once.

Does it work with a large list?

Yes, up to 5 MB of text in one go. The list is processed on your device and never uploaded.

How do I remove duplicates and sort at the same time?

Choose A to Z (or Z to A) in the Sort menu. The unique lines are sorted in natural order, so item2 comes before item10. On a Unix command line, sort -u does the same job.

How do I ignore case when removing duplicates?

Tick Ignore case. Then Apple, apple and APPLE count as one line, and the Keep setting decides which spelling is shown: the first occurrence or the last.

Why are two lines that look identical not removed?

They differ by something you cannot see, usually a trailing space, a tab, or different capitalisation. Make sure Trim spaces is ticked and, if capitalisation is the cause, tick Ignore case.

What is the difference between this and the uniq command?

The uniq command only compares adjacent lines, so the input must be sorted first. This tool finds repeated lines anywhere in the list and can keep the original order. It does not show counts per line.

Can I get only the lines that appear once?

Yes. Set Show to "Only lines that never repeat". Lines that occur two or more times are left out entirely, including their first copy.

Advertisement