Text Tools
Remove duplicate lines.
Paste a list and get it back with repeated lines removed. Choose whether case and spacing matter, keep the first or last copy, sort the result, or show only the lines that repeat.
Input
Result
Paste a list to remove its duplicates.
About this duplicate line remover
Each line is compared with every other line, and repeated lines are removed so every value appears once. It works on emails, URLs, keywords, SKUs, log lines and any other list that has one item per line.
Worked example
apple
banana
Apple
banana
cherry
With the defaults the output is apple, banana, Apple, cherry (one per line): "Apple" and "apple" are different until you tick Ignore case, and then the first spelling wins. Switching Show to "Only lines that repeat" returns just banana.
Options
- Keep first or last: the copy you keep decides both which spelling survives (with ignore case on) and where the line sits in the output.
- Trim spaces: "a" and "a " count as the same line, and the output is trimmed. Leave it off when trailing whitespace matters, for example in code.
- Sort: A to Z uses natural order, so
item2comes beforeitem10. Length sorts shortest first. Shuffle randomises the order. - Only duplicates / only unique: find the values that repeat, or the ones that never do, instead of deduplicating.
Limits
Matching is exact after the options above; similar but not identical lines (a typo, different punctuation) are kept. Input is limited to 5 MB. Processing runs in your browser.
Why duplicates appear, and which option fixes which kind
Duplicate lines creep in when lists are merged, exported twice or copied from several sources: email addresses collected from three campaigns, keyword lists exported from different tools, URLs pulled from several sitemaps, or log lines repeated by a retry loop. Not every duplicate looks identical, so the options matter more than the button.
- Exact duplicates (same characters) are removed by default.
- Case differences such as
Appleandappleneed Ignore case. Use it for lists, such as names or keywords, where capitalisation varies only because of how the data was typed. - Stray spaces make
aandalook different to a computer even though they look the same to you. Trim spaces is on by default for this reason. - Blank lines are removed by default. Turn that off if the blank lines are meaningful, for example paragraph breaks in a poem.
Worked example with numbers
Take a ten-line keyword list: seo tools, SEO Tools, seo tools (trailing space), keyword research, keyword research, backlinks, an empty line, backlinks, rank tracker, seo tools. With the defaults (trim on, ignore case off, empty lines removed) you get five lines: seo tools, SEO Tools, keyword research, backlinks and rank tracker. The trailing-space copy and the second plain seo tools merge with the first, but SEO Tools survives because case matters. Tick Ignore case and the result drops to four lines. Switch Show to "Only lines that repeat" and you get the three values that occurred more than once: seo tools, keyword research and backlinks. "Only lines that never repeat" returns SEO Tools and rank tracker, the two values that occur once.
Keep first or keep last?
Keeping the first occurrence preserves the order in which each value first appeared, which is what most people want. Keeping the last occurrence matters when later lines are corrections or newer entries, for example a log where the latest status for an ID is the one you care about. When Ignore case is on, this choice also decides which spelling survives: keep first returns Apple from Apple, apple, keep last returns apple.
Sorting and what "natural" order means
With Sort set to A to Z, items are put in natural order, so item2 comes before item10 instead of after it, as a plain character-by-character sort would put it. Z to A reverses that, By length puts the shortest lines first and Shuffle randomises the order. If you only need a sorted, deduplicated list on a Unix command line, sort -u does both: POSIX defines its -u option as suppressing all but one in each set of lines having equal keys, and -f folds lower case to upper case so case is ignored.
How this compares with the uniq command
The POSIX uniq utility compares adjacent lines only: the specification says repeated lines are not detected if they are not adjacent, so people sort first and pipe into uniq. uniq -d prints only repeated lines and uniq -u only lines that are not repeated, and uniq -c prefixes each line with a count. This page does not require sorted input and keeps the original order by default, and the "Only lines that repeat" and "Only lines that never repeat" choices correspond to the -d and -u behaviours. It does not show per-line counts; if you need frequencies, use sort | uniq -c in a terminal.
Common mistakes
- Expecting near-duplicates to merge.
colourandcolor, or two URLs that differ by a trailing slash, are different lines. Normalise them first, for example with the case converter, then deduplicate. - Trimming when whitespace is meaningful. In code or fixed-width data, leading spaces carry meaning. Turn trimming off.
- Pasting a comma-separated row. The tool works one item per line. Convert commas to line breaks first, or use the list to comma separated tool after cleaning.
- Losing the original list. The result replaces nothing, but copy your source somewhere before you overwrite it with the output.
Limits
Input is limited to 5 MB. Matching is exact once the chosen options are applied, so the tool will not detect typos or fuzzy matches. It compares whole lines, not words inside them. To see what changed between two versions of a list, use the text diff. Everything runs in your browser.
Frequently asked questions
How do I remove duplicate lines but keep the original order?
Leave Sort on "Keep original order". The first occurrence of each line stays where it was and later copies are removed.
How do I find which lines are duplicated?
Set Show to "Only lines that repeat". The status line also counts how many distinct lines appeared more than once.
Does it work with a large list?
Yes, up to 5 MB of text in one go. The list is processed on your device and never uploaded.
How do I remove duplicates and sort at the same time?
Choose A to Z (or Z to A) in the Sort menu. The unique lines are sorted in natural order, so item2 comes before item10. On a Unix command line, sort -u does the same job.
How do I ignore case when removing duplicates?
Tick Ignore case. Then Apple, apple and APPLE count as one line, and the Keep setting decides which spelling is shown: the first occurrence or the last.
Why are two lines that look identical not removed?
They differ by something you cannot see, usually a trailing space, a tab, or different capitalisation. Make sure Trim spaces is ticked and, if capitalisation is the cause, tick Ignore case.
What is the difference between this and the uniq command?
The uniq command only compares adjacent lines, so the input must be sorted first. This tool finds repeated lines anywhere in the list and can keep the original order. It does not show counts per line.
Can I get only the lines that appear once?
Yes. Set Show to "Only lines that never repeat". Lines that occur two or more times are left out entirely, including their first copy.
Sources
- The Open Group Base Specifications: uniq (retrieved 2026-10)
- The Open Group Base Specifications: sort (retrieved 2026-10)