Deduplicate lines
Remove duplicate lines, keep first occurrence or count occurrences
Enter one item per line and keep first or last occurrences. To inspect frequency, switch the output mode to count plus content.
Example
Input
apple pear apple
Example result
apple pear
Keep-first produces this order; keep-last produces pear, apple.
Technical details & limits
A Map uses each processed whole line as a key; matching is exact, not fuzzy. First mode preserves first-occurrence order, while last mode uses last-occurrence positions. Case-insensitive mode changes comparison keys but keeps the selected line's casing. Empty lines also participate.
About this tool
Remove duplicate whole lines, keeping the first or last occurrence, with optional counts. Case-insensitive comparison and trimming help clean lists, log keys and exported text.
Use cases
- Deduplicate a merged list of records.
- See which request paths show up most often in a log.
How to use
- Paste the text.
- Choose match options and output mode. Result updates live.
Notes
Rows are compared whole, with leading and trailing whitespace trimmed first by default. Order follows first appearance, and any difference inside a line — even one space — makes it a different line.
FAQ
- What is the difference between keeping first and last duplicate lines?
- Keeping the first occurrence orders results by first appearance, while keeping the last orders them by the final retained positions. For A, B, A across 3 lines, the first mode returns A, B and the last returns B, A; both retain 2 lines. Choose the option that fits your list's meaning, particularly when later entries are intended to replace earlier records rather than merely repeat them.
- Does case-insensitive deduplication change the retained line's spelling?
- Case-insensitive deduplication changes the comparison key but preserves the spelling of the retained original line. For example, A and a are 2 input lines that become 1 retained entry. Keeping the first or last occurrence determines which spelling survives. Empty lines also participate in comparison. Check the chosen retention and whitespace options before copying the output if original capitalization or blank separators carry meaning in your data.
- Are duplicate line counts based on whole lines or partial keywords?
- Duplicate counts compare the entire processed line rather than partially matching a keyword. For example, abc1 and abc2 are 2 distinct entries, while the same abc1 appearing twice receives a count of 2. Whitespace trimming and case-insensitive options can alter the comparison key. Interpret the counts using those settings, and do not treat the result as a fuzzy matching or semantic record reconciliation process for similar-looking entries.
Related tools
Sort lines
Text diff (by line)