Remove Duplicate Lines

When you need to remove duplicate lines

Lists collected from several sources almost always contain repeats: email addresses exported from two CRMs, keywords merged from multiple research tools, SKUs copied from different sheets, log entries, or URLs from a crawl. Cleaning them by hand is slow and error-prone. Paste the list here and the unique version appears immediately, with a count of how many lines were removed.

How the options work

• Ignore case treats Delhi, delhi and DELHI as the same line. The first spelling that appears is the one that is kept.
• Trim whitespace removes spaces and tabs at the start and end of each line before comparing, so apple (with a trailing space) and apple are recognised as duplicates. Spaces in the middle of a line are left alone.
• Remove empty lines drops blank lines from the output. Turn it off if you use blank lines to separate groups.
• Sort orders the result alphabetically A → Z or Z → A. Sorting is “natural”, so item2 comes before item10, and it is accent- and case-aware. Leave it on Keep original order to preserve the sequence in which lines first appeared.

The counters show input lines, unique lines and removed lines. A trailing newline at the end of the pasted text is not counted as a line.

Example

Input:

Mumbai
Delhi
mumbai
Bengaluru
Delhi

Chennai

With Ignore case, Trim whitespace and Remove empty lines enabled, the output is:

Mumbai
Delhi
Bengaluru
Chennai

Seven input lines, four unique, three removed (one blank, two duplicates).

Practical uses

• Email lists: dedupe before importing into Mailchimp or Zoho Campaigns to avoid sending twice and to keep your subscriber count honest. Enable Ignore case because email addresses are not case-sensitive.
• SEO keyword research: merge exports from Google Keyword Planner, Ahrefs and Semrush, dedupe, then sort A → Z to spot near-duplicates such as singular and plural forms.
• Code and config files: remove repeated imports, .gitignore entries or hosts-file lines.
• Data entry: clean up pin codes, GST numbers or invoice IDs before pasting into Excel; it is quicker than building a Remove Duplicates step in the spreadsheet.

Tips

• Lines are compared as whole strings. Delhi, India and Delhi are different lines even though one contains the other.
• If your list is comma-separated rather than one item per line, first replace the commas with line breaks in a text editor, or use a find-and-replace tool, then paste here.
• Very large lists (tens of thousands of lines) are fine; the work is done in memory in your browser and nothing is uploaded.
• Copy the cleaned result with the Copy result button, or select the output box and copy manually if clipboard permission is blocked.

Frequently asked questions