All Tools

The tool splits your input at every line break into an ordered list of lines. It then walks that list once, building a comparison key for each line and keeping a record of the keys it has already seen. A line whose key has appeared before is treated as a duplicate.

The options change how the key is built, not how the output is printed. With trim whitespace enabled, leading and trailing spaces and tabs are stripped before the key is made, so an indented copy of a line matches the unindented original. With ignore case enabled, the key is lowercased, so Apple and apple collapse into one entry. In both cases the line that survives is printed exactly as you typed it, with its original spacing and capitalisation intact.

The keep-which-occurrence setting decides which copy survives. Keeping the first preserves the original order of appearance, which is what you usually want for lists. Keeping the last is useful for logs and configuration files where a later entry supersedes an earlier one. Remove empty lines drops blanks entirely, and sorting is applied at the end, after deduplication.

Take this input, seven lines in total: apple, Banana, apple, cherry, an empty line, banana, Cherry.

Run it with ignore case on, remove empty lines on, keep first occurrence, and sorting off. The blank line is dropped, leaving six lines to compare. The keys become apple, banana, apple, cherry, banana, cherry. Walking through in order: apple is new and kept; Banana is new and kept; the second apple matches a key already seen and is removed; cherry is new and kept; banana matches Banana's key and is removed; Cherry matches cherry's key and is removed.

The output is three lines: apple, Banana, cherry. Note that Banana keeps its capital B because the original text is preserved; only the comparison was case-insensitive. The statistics show four lines removed in total, three of them duplicates and one a blank.

Run the same input with ignore case off and the result is five lines, because Banana and banana are then distinct, as are cherry and Cherry. Switching keep-last on instead returns apple, cherry, banana and Cherry in their later positions.

Case sensitivity is the setting that catches people out. Email addresses should almost always be compared case-insensitively, since the domain part is not case-sensitive and most providers ignore case in the local part too. Code identifiers, passwords and case-sensitive file paths on Linux should not be, because Config.yml and config.yml are genuinely different files.

Invisible whitespace is the second trap. Lines copied from a web page or a PDF frequently carry trailing spaces or non-breaking characters, so two apparently identical lines fail to match. Enabling trim whitespace resolves the ordinary cases of leading and trailing spaces and tabs.

Be careful with structured data. A CSV row is only a duplicate if the entire line matches character for character, so two records for the same person with columns in a different order, or with one extra space after a comma, will not be detected as duplicates. Deduplicate CSV files by a key column in a spreadsheet instead.

Finally, remember that sorting is applied after deduplication and reorders the whole output. If the original sequence carries meaning, such as in a log, leave sorting off.