File tools · CSV
Remove duplicates from a CSV by the columns you choose.
Find repeated records by whole row or by key columns like email or customer ID. See what was removed, keep the first or last copy, and download the removed rows too.
Whole row or key columns: two rows are duplicates when the columns you tick match after the cleanup options you choose. Nothing is uploaded.
How duplicates are decided
Each row gets a comparison key built from the columns you chose. With “every column” the key is the whole row; with key columns it is only those cells, such as email, or first name + last name + birth date together. Before comparing, the cleanup options are applied to the key only: trimming spaces, folding case, collapsing internal runs of spaces. The row you keep is written out exactly as it was in the file; the options never rewrite your data.
Keeping the first occurrence leaves the earliest row in place. Keeping the last puts the latest version of each record where the first one stood, which is handy when later rows are corrections. Multi-column keys are compared as a list, not as joined text, so x,y + z never collides with x + y,z.
Your data stays on your device
Rows are compared in memory inside this tab. The frame has no permission to open network connections and no analytics or advertising script runs inside it, so what is in your file is visible to you and nobody else. Customer lists, mailing lists and accounting exports are exactly the files worth cleaning without handing them to a third party.
Worked example
A mailing list has the rows 1,A@x.com, 2,a@x.com (with a trailing space), 3,b@x.com, a blank row, 4,B@x.com and a second copy of 1,A@x.com. With whole-row matching only the exact second copy of row 1 is removed. Switch to key column email with case and spaces ignored and rows 2, 4 and the repeated row 1 go too, leaving ids 1 and 3 and the blank row. Tick “drop blank rows” and the blank row goes as well. The page lists the removed rows and offers them as a separate download, so you can see what a rule caught before you rely on it.
Should I ignore case in emails?
Usually, but not always. RFC 5321 allows the part of an address before the @ to be case-sensitive, yet in practice nearly all mail providers treat Ann@x.com and ann@x.com as one mailbox. Ignoring case is the safe default for a mailing list; for other identifiers such as case-sensitive codes or file names, leave it off.
Size and speed
Duplicate detection uses a hash lookup, so the time grows in step with the number of rows, not with its square. In our automated test, 100,000 rows with 10 columns were checked against a key column in well under a second, and in a real browser the whole load-and-compare took under two seconds. Memory is the practical limit: the file, its parsed rows and the result all sit in the tab at once.
Common mistakes
- Using whole-row matching on records with a changing field. If every row has a timestamp or row number, no two rows are ever identical. Choose the key columns instead.
- Keeping the wrong copy. The tool does not pick the “most complete” row. It keeps the first or last one it meets, so sort the file first if you need a particular copy to win.
- Expecting fuzzy matching.
Jon SmithandJohn Smithare different. Only exact matches, after the cleanup options, count. - Forgetting the header. With the header box ticked the first row is never treated as data and never removed.
What this tool does not do
It does not merge conflicting values from two duplicate rows, match similar-looking names, or remove duplicates across several files at once; merge the files first with the CSV merger. For a plain list of lines rather than CSV records, use remove duplicate lines.
Frequently asked questions
Is my CSV sent to a server?
No. The file is read and compared in your browser tab only. The cleaned file and the removed-rows file are created locally when you press download.
Can I remove duplicates based on one column, such as email?
Yes. Choose “The ticked key columns match” and tick the column or columns that identify a record. Other columns are ignored when deciding what is a duplicate.
Which row is kept?
The first occurrence by default, left in its original position. Choose “last occurrence” to keep the latest copy instead, which then takes the place of the first.
Do the case and space options change my data?
No. They only affect how rows are compared. The row that is kept is written exactly as it appeared in the file.
Can I see which rows were removed?
Yes. The result shows the first ten removed rows and a button downloads all of them as a separate CSV, with the same header.
How big a file can it handle?
There is no built-in cap; rows are held in memory. In our automated test 100,000 rows with a key-column check finished in a fraction of a second.
Does it find near-duplicates like “Jon” and “John”?
No. Matching is exact after the trim, case and space options. Fuzzy matching needs human review and is not attempted.
Sources
- RFC 4180: Common Format and MIME Type for CSV Files (Y. Shafranovich, October 2005) — the record, quoting and line-break rules the parser follows
- RFC 5321: Simple Mail Transfer Protocol — that the local part of an email address may be case-sensitive
- Microsoft: Find and remove duplicates — what Excel's own duplicate tools offer, if you prefer to work inside a spreadsheet
Review before you delete
Always check the removed-rows file against your original before replacing it. A key that looks unique to you may not be unique in the data.