Remove Duplicate Lines_
Paste a list and get it back without repeats. Keep the first copy or the last, show only the lines that repeat, or only the ones that appear exactly once, with optional counts and sorting. Nothing is reordered unless you ask.
The part worth having is the diagnosis. When two lines look identical on screen and the tool keeps both, it tells you why — a trailing space, a non-breaking space pasted from a web page, a zero-width character, two Unicode spellings of the same accent — and which switch will fix it.
Some lines look identical but are not equal. Each row says why, how many more would collapse, and which switch above fixes it.
- Input
- Lines pasted or typed. Every line-ending convention is read the same way, so the file's origin makes no difference.
- Output
- The surviving lines, with optional occurrence counts, ready to copy or download.
- Processing
- Compared in this tab under whichever rules you pick, then reported: how many were removed and how many distinct values there were.
- Limits
- Comparison is line by line. Reordering belongs to a sort tool; comparing two different texts belongs to a diff.
- The failure this exists for
- Two lines that read the same and are not equal. A trailing space, a non-breaking space, a zero-width character or a different Unicode composition — none of them visible in any font, all of them enough to defeat a match.
When two identical lines are not duplicates
The tool is not broken, the text is
The commonest complaint about every deduplicator is that it missed some. Almost always both lines really are different, by something with no glyph. A trailing space is the usual one, and a monospace font shows you nothing at all. Next is the non-breaking space, which arrives whenever text is copied out of a web page, a PDF or Word: John Smith and John Smith are not the same string if the gap in one is U+00A0. Then zero-width characters and stray byte-order marks, which are invisible by definition. This page checks for each of those and names the one it found.
Two ways to spell an accent
Unicode can write café as four characters ending in U+00E9, or as five ending in a plain e followed by a combining acute accent. Both render identically in every font. macOS filesystems have historically preferred the decomposed form and most other things prefer the composed one, so a list assembled from two sources can contain both spellings of the same word. Normalising to NFC before comparing makes them equal, which is what the Normalise Unicode switch does.
Why uniq seems to do nothing
The Unix uniq only collapses runs of identical adjacent lines. Run it on an unsorted file and it looks broken, which is exactly why the idiom is sort file | uniq rather than uniq file. Notepad++ behaves the same way — its built-in command is called Remove Consecutive Duplicate Lines and means it. Both behaviours are here: leave Adjacent only off for a whole-file pass that preserves order, or switch it on to reproduce what the shell and the editor do.
Which copy survives can change the answer
Keeping the first occurrence is the default because it preserves the order you had. But in anything where a later entry overrides an earlier one — a config file, an environment list, a log of state changes — the last copy is the one that matters and keeping the first quietly gives you stale values. The surviving lines stay in their original relative order either way, so switching to Last changes which text you keep without scrambling the list.
Paste, choose the rules, take the result
- 01Paste your lines on the left. The result appears on the right immediately, and the status line says how many were removed out of how many.
- 02If lines you expected to collapse did not, read the panel underneath — it names the invisible difference and the switch that neutralises it.
- 03Pick what to keep: one of each, only the values that repeat, or only the values that appear exactly once. Add counts if you want to know how often each occurred.
- 04Copy or download the result. Nothing is sorted unless you ask, so the order you pasted is the order you get back.
Four lists and what to do with them
An email list merged from two exports
One export trimmed its fields and the other did not.
Ignore outer space
The pairs finally collapse
Finding what appears more than once
You want the repeats themselves, not the deduplicated list.
Only the repeats
sort file | uniq -d
A config where the last value wins
Keeping the first copy would preserve the overridden value.
Last
The effective values, in place
Counting how often each line occurs
A frequency list without leaving the browser.
Prefix counts, Sort Z to A
sort | uniq -c | sort -rn
Why two identical-looking lines are not equal
| Cause | Where it comes from | What fixes it here |
|---|---|---|
| Trailing whitespace | Hand editing, wrapped exports, anything that pads a column | Ignore outer space |
| Leading whitespace | Indented lists, copied code, pasted table cells | Ignore outer space |
| Doubled inner spaces | Manual alignment, or joining fields with an extra separator | Collapse inner space |
| A non-breaking space (U+00A0) | Copying from a web page, a PDF, Word or a spreadsheet | Collapse inner space, which folds unusual spaces too |
| Zero-width characters, byte-order marks | UTF-8 files with a BOM, invisible marks in copied rich text | Nothing here removes them — strip them first |
| Different Unicode composition | Lists merged from macOS and elsewhere; é written two ways | Normalise Unicode |
| Letter case | Free-text entry, logs, anything typed by more than one person | Turn off Case sensitive |
Line endings are deliberately missing from this list. Every convention is treated as a break before comparison begins, so whether a file came from Windows, Unix or a classic Mac cannot be the reason two lines fail to match.
How to do it in the tool you already have open
| Where | How | Worth knowing |
|---|---|---|
| Notepad++ | Edit → Line Operations → Remove Consecutive Duplicate Lines | Consecutive only, so sort first (Edit → Line Operations → Sort Lines Lexicographically) unless the duplicates are already adjacent. |
| Notepad++ (any order) | TextFX → TextFX Tools → Sort outputs only UNIQUE lines | Needs the TextFX plugin, and it sorts as a side effect — the original order is gone. |
| VS Code | F1 → "Delete Duplicate Lines" | Built in since 1.62 and keeps the original order. There is no keybinding by default. |
| Sublime Text | Edit → Permute Lines → Unique | Keeps the first occurrence and preserves order. |
| Shell, order preserved | awk '!seen[$0]++' file | The idiomatic one. Keeps first occurrences in their original order and needs no sort. |
| Shell, sorted | sort -u file | Sorts and deduplicates in one pass. Output order is the sort order, not the input order. |
| Shell, uniq | sort file | uniq | uniq only collapses ADJACENT duplicates, which is why the sort is not optional. `uniq -d` shows only repeats, `uniq -u` only singles, `uniq -c` prefixes counts. |
| Excel | Data → Remove Duplicates | Operates on rows and edits in place. It also trims nothing, so trailing spaces defeat it exactly as they do everywhere else. |
Most searches for this are asking which keystroke, not which website. These are the answers; the tool above is for when the list is already in your clipboard or the duplicates are not collapsing.
Getting a clean list
- Read the diagnosis panel before reaching for a switch. It tells you which difference is actually present rather than making you try each option in turn.
- Turn on Prefix counts with Sort Z to A to get a frequency list — the most common lines first, which is usually what you wanted from a duplicate check anyway.
- Use "Only the one-offs" to find the entries that appear in one source and not the other after concatenating two lists.
- Switch Copy to Last for anything where a later line overrides an earlier one; the default keeps the first, which is the wrong value in a config.
- Leave Sort on "Leave order" unless you need it. Order is information, and a deduplicator that silently sorts destroys it.
- For counting lines, blanks and longest lines rather than removing anything, the line counter diagnoses without changing the text.
Where a duplicate check goes wrong
Invisible characters are the usual reason it "missed some"
A trailing space, a non-breaking space, a zero-width joiner or a byte-order mark all defeat an exact match and none of them is visible. That is not a bug in whatever tool you tried; the two lines genuinely differ. This page reports which of those is present instead of leaving you to guess.
Zero-width characters are reported but not removed
There is no switch here that strips them, deliberately. Deleting characters from inside your text is a bigger and more dangerous edit than ignoring whitespace at the edges, and a tool that did it quietly could change data you meant to keep. The panel names them so you can remove them where they came from.
Ignoring case or space changes what matches, not what you get back
The comparison is normalised; the output is not. If you deduplicate "ALPHA" and "alpha" with case ignored, the surviving line is whichever came first, spelled exactly as it was. That is usually what you want and is worth knowing before you paste the result somewhere case matters.
Adjacent-only will look like it did nothing
On an unsorted list it is supposed to. It collapses runs, not scattered repeats, which is the behaviour of uniq and of the built-in Notepad++ command. If you meant "remove every duplicate", leave that switch off.
Sorting is not the default and should not be
Many tools sort as a side effect of deduplicating, which destroys the order of a list where order carries meaning — a playlist, a log, a priority list. Here the output keeps its original order unless you choose otherwise.
Comparison rules and what gets reported
- Line splitting
- Delegated to the shared module, so the rules are exactly those the line counter publishes in its own table — including how a document that ends in a newline is counted. Two tools disagreeing about what a line is would be worse than either being wrong alone
- Comparison key
- Built in a fixed order: Unicode normalisation, then trimming, then whitespace collapsing, then case folding. Order matters because normalisation can change what counts as whitespace at the edges
- Whitespace classes
- The space rules match U+0020, tab, vertical tab, form feed and the Unicode spaces U+00A0, U+1680, U+2000–U+200A, U+202F, U+205F, U+3000. Zero-width characters are deliberately excluded from that set — JavaScript counts U+FEFF as whitespace, which would report a byte-order mark as a leading space
- Modes
- One of each, only values occurring more than once (
uniq -d), or only values occurring exactly once (uniq -u). Counts can be prefixed asuniq -cdoes, padded so the text stays in a column - Adjacent only
- Reproduces
uniq— runs collapse, scattered repeats do not. In this mode a prefixed count is the length of the run, not the total for the file, which is also what uniq reports - Diagnosis
- Each cause is tested independently against exact equality and reports how many extra lines would collapse if only that one were neutralised, so the number names a single switch rather than an unattributable total
- Order
- Preserved unless a sort is chosen. Sorting compares by the same key as deduplication, so an ignored case does not scatter the result
- Scope
- One text, line by line. Comparing two different texts belongs to text compare and diff checker
- Network
- None from tool code. A test sweep calls every function this page uses with
fetchandXMLHttpRequestreplaced by stubs that throw, so a stray request fails the build instead of shipping. Disconnect from the network and the page still works.
Questions about removing duplicate lines
Why did it not remove lines that look identical?
Because they are not identical. The usual culprits are a trailing space, a non-breaking space pasted from a web page or PDF, a zero-width character, or the same accented letter written two different ways in Unicode. All are invisible in every font. Paste the list and the panel under the output names which one is present and which switch neutralises it.
How do I remove duplicate lines in Notepad++?
Edit → Line Operations → Remove Consecutive Duplicate Lines. The word consecutive is literal — it only collapses runs, so sort first with Edit → Line Operations → Sort Lines Lexicographically unless your duplicates are already adjacent. With the TextFX plugin, TextFX → TextFX Tools → "Sort outputs only UNIQUE lines" handles any order, but it sorts as a side effect.
How do I do it on the command line?
For order-preserving removal, awk '!seen[$0]++' file is the idiom. For sorted output, sort -u file does both in one pass. Plain uniq only collapses adjacent duplicates, which is why you almost always see it written as sort file | uniq; add -d for only the repeats, -u for only the one-offs, or -c to prefix counts.
Does it keep the first or the last copy?
The first, by default, because that preserves the order you pasted. Switch Copy to Last when a later line is meant to override an earlier one — a config file, an environment list, a sequence of state changes. Either way the survivors stay in their original relative order rather than being reversed.
Will it reorder my list?
Not unless you ask. Sorting is a separate control set to "Leave order" by default, because plenty of lists carry meaning in their order and a tool that quietly sorts destroys it. When you do sort, the comparison uses the same key as the deduplication, so ignoring case does not scatter the result.
Can it remove the invisible characters it finds?
It reports them and does not strip them, on purpose. Ignoring whitespace at the edges of a line is a comparison rule; deleting characters from the middle of your text is an edit, and a tool that did that silently could alter data you meant to keep. The panel names what it found so you can fix it at the source.
Is my list uploaded anywhere?
The work is JavaScript running in this tab. Every function it calls is covered by a test that stubs fetch and XMLHttpRequest to throw, so a request that slipped in would break the build rather than reach a server — and you can confirm it for yourself by disconnecting and carrying on.