What does the Duplicate Line Remover do?
It takes a list with one item per line and returns each item only once. Lists collect duplicates easily: email addresses merged from two exports, URLs crawled from several sitemaps, keywords pooled from different research tools, tags or product SKUs pasted from spreadsheets. Removing them by hand is slow and error-prone once a list has more than a few dozen lines.
Unlike a plain "unique" filter, this tool lets you decide what counts as the same line. Ignore case treats Example.com and example.com as duplicates; ignore leading/trailing spaces catches lines that differ only by invisible indentation. You choose whether the first or last occurrence survives, what happens to empty lines, and whether the result is sorted.
The original order is preserved unless you ask for sorting, and a second panel lists every line that was repeated with how many times it appeared — useful for auditing where the duplicates came from.
How to use the Duplicate Line Remover
- Paste your list into the input box, one item per line, or click Sample to load a list of URLs with duplicates.
- Choose the comparison options: ignore case, ignore surrounding spaces, and how empty lines are handled.
- Pick whether to keep the first or last occurrence, and optionally sort the output A → Z.
- Copy or download the unique lines. Use Copy list under Removed duplicates to get each repeated line with its count.
Example
With Ignore case on, keeping the first occurrence:
Input: Output:
newsletter newsletter
WordPress WordPress
seo seo
wordpress
SEO
newsletterFirst or last occurrence?
Keeping the first occurrence is the usual choice and matches what spreadsheet "remove duplicates" features do. Keeping the last is useful when later lines are more recent — for example a log or an export where an updated entry was appended to the end — because the surviving line stays at its latest position.
With Ignore case on, the line that is kept is printed exactly as it appears at that position, so choose first or last depending on which spelling you want to keep.
Frequently Asked Questions
Does it change the order of my lines?
No. Unique lines stay in their original order unless you tick Sort output A → Z. To sort in other ways — by length, numerically or randomly — use Sort Lines, which can also remove duplicates.
How are empty lines handled?
By default every empty line is kept so paragraphs stay separated. Choose Remove empty lines to drop them all, or Treat like other lines to keep only the first empty line.
Can it handle large lists?
Yes. De-duplication uses a hash set, so lists of hundreds of thousands of lines are processed in well under a second on a typical laptop.
Why are two lines that look identical not treated as duplicates?
They probably differ by an invisible character such as a non-breaking space, a zero-width space or a different apostrophe. Run the list through the Text Cleaner first, then remove duplicates.
How do I count duplicates without removing them?
The Line Counter reports unique and duplicate line counts, and the Removed duplicates panel here shows how often each repeated line appeared.