Deduplicate lines
Remove duplicate lines, keep first occurrence or count occurrences
Enter one item per line and keep first or last occurrences. To inspect frequency, switch the output mode to count plus content.
Example
Input
apple pear apple
Example result
apple pear
Keep-first produces this order; keep-last produces pear, apple.
Technical details & limits
A Map uses each processed whole line as a key; matching is exact, not fuzzy. First mode preserves first-occurrence order, while last mode uses last-occurrence positions. Case-insensitive mode changes comparison keys but keeps the selected line's casing. Empty lines also participate.
About this tool
Remove duplicate whole lines, keeping the first or last occurrence, with optional counts. Case-insensitive comparison and trimming help clean lists, log keys and exported text.
Use cases
- Deduplicate a merged list of records.
- See which request paths show up most often in a log.
How to use
- Paste the text.
- Choose match options and output mode. Result updates live.
FAQ
- Why are two apparently identical lines not merged?
- They may differ in whitespace, case or Unicode composition. Enable trimming or case-insensitive matching as needed. Unicode normalization is not automatic.
- Can I see how often each line occurs?
- Select count output to get a count, tab and line text. Case and trimming rules still apply.
- Does deduplication alphabetize the result?
- No. Output follows the positions of the kept first or last occurrences. Use line sorting separately when needed.
- Does ignoring case make the result lowercase?
- No. It only changes comparison keys. Output keeps the case of the chosen first or last occurrence; enabled trimming also removes leading and trailing whitespace.
Related tools
Sort lines