🗑️
Duplicate Line Remover
Paste your text, configure options, and remove duplicate lines instantly. Supports case-insensitive matching, whitespace trimming, and more.
⚠️
⚙️ Processing Options
Ignore Case
"Hello" and "hello" treated as duplicates
Trim Whitespace
Strip leading/trailing spaces before comparing
Remove Empty Lines
Delete blank lines from the output
Sort Output
Alphabetically sort the final lines
💡 Tips
Each line is one unit. Lines are compared character-by-character unless options are on.
Ignore Case + Trim gives the most aggressive deduplication.
"Keep Last" is useful for config files where the last value overrides earlier ones.
Input Text
🗑️
Configure options and click Remove Duplicates
Output and statistics will appear here
Processing…
Input
total lines
Removed
duplicates
Blanks
removed
Output
unique lines
🗑️
✅
↕ Sorted
Aa case-insensitive
Cleaned Output
✅
No output — all lines were removed
Saved
of lines
How duplicate detection works
The tool splits your input at every line break into an ordered list of lines. It then walks that list once, building a comparison key for each line and keeping a record of the keys it has already seen. A line whose key has appeared before is treated as a duplicate.
The options change how the key is built, not how the output is printed. With trim whitespace enabled, leading and trailing spaces and tabs are stripped before the key is made, so an indented copy of a line matches the unindented original. With ignore case enabled, the key is lowercased, so Apple and apple collapse into one entry. In both cases the line that survives is printed exactly as you typed it, with its original spacing and capitalisation intact.
The keep-which-occurrence setting decides which copy survives. Keeping the first preserves the original order of appearance, which is what you usually want for lists. Keeping the last is useful for logs and configuration files where a later entry supersedes an earlier one. Remove empty lines drops blanks entirely, and sorting is applied at the end, after deduplication.
The options change how the key is built, not how the output is printed. With trim whitespace enabled, leading and trailing spaces and tabs are stripped before the key is made, so an indented copy of a line matches the unindented original. With ignore case enabled, the key is lowercased, so Apple and apple collapse into one entry. In both cases the line that survives is printed exactly as you typed it, with its original spacing and capitalisation intact.
The keep-which-occurrence setting decides which copy survives. Keeping the first preserves the original order of appearance, which is what you usually want for lists. Keeping the last is useful for logs and configuration files where a later entry supersedes an earlier one. Remove empty lines drops blanks entirely, and sorting is applied at the end, after deduplication.
Worked example on a seven-line list
Take this input, seven lines in total: apple, Banana, apple, cherry, an empty line, banana, Cherry.
Run it with ignore case on, remove empty lines on, keep first occurrence, and sorting off. The blank line is dropped, leaving six lines to compare. The keys become apple, banana, apple, cherry, banana, cherry. Walking through in order: apple is new and kept; Banana is new and kept; the second apple matches a key already seen and is removed; cherry is new and kept; banana matches Banana's key and is removed; Cherry matches cherry's key and is removed.
The output is three lines: apple, Banana, cherry. Note that Banana keeps its capital B because the original text is preserved; only the comparison was case-insensitive. The statistics show four lines removed in total, three of them duplicates and one a blank.
Run the same input with ignore case off and the result is five lines, because Banana and banana are then distinct, as are cherry and Cherry. Switching keep-last on instead returns apple, cherry, banana and Cherry in their later positions.
Run it with ignore case on, remove empty lines on, keep first occurrence, and sorting off. The blank line is dropped, leaving six lines to compare. The keys become apple, banana, apple, cherry, banana, cherry. Walking through in order: apple is new and kept; Banana is new and kept; the second apple matches a key already seen and is removed; cherry is new and kept; banana matches Banana's key and is removed; Cherry matches cherry's key and is removed.
The output is three lines: apple, Banana, cherry. Note that Banana keeps its capital B because the original text is preserved; only the comparison was case-insensitive. The statistics show four lines removed in total, three of them duplicates and one a blank.
Run the same input with ignore case off and the result is five lines, because Banana and banana are then distinct, as are cherry and Cherry. Switching keep-last on instead returns apple, cherry, banana and Cherry in their later positions.
Getting the options right for your data
Case sensitivity is the setting that catches people out. Email addresses should almost always be compared case-insensitively, since the domain part is not case-sensitive and most providers ignore case in the local part too. Code identifiers, passwords and case-sensitive file paths on Linux should not be, because Config.yml and config.yml are genuinely different files.
Invisible whitespace is the second trap. Lines copied from a web page or a PDF frequently carry trailing spaces or non-breaking characters, so two apparently identical lines fail to match. Enabling trim whitespace resolves the ordinary cases of leading and trailing spaces and tabs.
Be careful with structured data. A CSV row is only a duplicate if the entire line matches character for character, so two records for the same person with columns in a different order, or with one extra space after a comma, will not be detected as duplicates. Deduplicate CSV files by a key column in a spreadsheet instead.
Finally, remember that sorting is applied after deduplication and reorders the whole output. If the original sequence carries meaning, such as in a log, leave sorting off.
Invisible whitespace is the second trap. Lines copied from a web page or a PDF frequently carry trailing spaces or non-breaking characters, so two apparently identical lines fail to match. Enabling trim whitespace resolves the ordinary cases of leading and trailing spaces and tabs.
Be careful with structured data. A CSV row is only a duplicate if the entire line matches character for character, so two records for the same person with columns in a different order, or with one extra space after a comma, will not be detected as duplicates. Deduplicate CSV files by a key column in a spreadsheet instead.
Finally, remember that sorting is applied after deduplication and reorders the whole output. If the original sequence carries meaning, such as in a log, leave sorting off.
Frequently Asked Questions
Open the file, copy its contents, paste them into the input box and run the tool. Choose whether to keep the first or last occurrence, and enable ignore case or trim whitespace if the repeats differ only in capitalisation or spacing. Copy the cleaned output back into a new file rather than overwriting the original.
Yes, unless you enable sorting. With keep first occurrence selected, each unique line stays in the position where it first appeared, so the sequence is preserved apart from the removed repeats. Selecting keep last occurrence moves each surviving line to its final position instead, which suits logs where later entries override earlier ones.
Yes. Turn on the ignore case option and lines are compared in lowercase, so Apple, APPLE and apple are treated as the same entry. The surviving line keeps the exact capitalisation you typed, because the lowercasing affects only the comparison. Leave the option off for case-sensitive data such as code identifiers or Linux file paths.
Yes, and it is one of the most common uses. Put one address per line, enable ignore case and trim whitespace, and switch on remove empty lines to drop the gaps that copying from a spreadsheet often introduces. Addresses that differ by an alias or a plus tag will remain, since those are genuinely different strings.
No. The splitting, comparison and output are all performed by JavaScript running in your browser, so the text never leaves your device. That makes the tool suitable for internal lists, log excerpts and anything else you would rather not paste into an online service that processes content on its own servers.