Visit Website →
← Back to blog

How to Remove Duplicate Lines From Any Text

November 10, 2026
How to Remove Duplicate Lines From Any Text

To remove duplicate lines, scan the text from top to bottom, keep the first copy of each line and drop every later copy. Online line-dedupe tools, a text editor's built-in command, one line of awk or a short Python script all do this. The part that takes care is deciding what counts as "the same line": capitalization, trailing spaces and blank lines all change the result. This guide covers each method and the traps in each, and ends with a way to check your output before you share it.

fixorahub.com
Clean Up Text in Your Browser
Use Fixora’s browser-based tools to clean up text and handle everyday file tasks, with processing kept local and no account required.
Visit

TL;DR

  • Exact deduplication keeps the first occurrence of each line and preserves the original order. Sorting first is faster to run but scrambles the order.
  • Lines that look identical often are not: a trailing space, a different capital letter or a Windows line ending makes them "different" to a tool.
  • uniq on its own only removes duplicates that sit next to each other. Use awk '!seen[$0]++' or sort -u for the whole file.
  • In a CSV, decide which columns define a duplicate before you delete anything. Whole-row comparison and key-column comparison give different results.
  • Keep the original, then compare it with the cleaned version and check the line count before you use the result.

Table of Contents

What counts as a duplicate line

A duplicate-line tool compares text strings, one per line. Two lines are duplicates when every character matches, from the first character to the last. That sounds obvious until you meet real data.

These pairs look identical on screen but are not identical to a strict tool:

Line ALine BWhy they differ
AppleappleCapitalization
appleapple One trailing space
appleapple followed by a tabInvisible tab character
apple (Windows file)apple (Unix file)Carriage return before the line break
appleappelTypo, a genuinely different line

Good dedupe tools expose options for exactly these cases: ignore case, trim whitespace, and sometimes sort the output. If a tool offers none of them, assume it is doing strict comparison.

The POSIX specification for uniq shows how this works in practice. It describes options to skip initial fields or characters when comparing lines, so the comparison can focus on the part of the line you care about, as the POSIX uniq page explains. The same idea applies in a browser tool: you choose what the comparison looks at.

Near-duplicates are not duplicates

Lines that are "almost" the same, such as John Smith and Smith, John, will never be removed by an exact-match tool. That is correct behavior. Merging them means deciding which version is right, which is a data-cleaning decision and not a text operation. Normalize the format first (case, spacing, punctuation), then remove duplicates. Most surprises come from skipping that step.

Blank lines

Blank lines are lines too. If your text has twenty empty lines, a strict dedupe keeps the first one and removes the other nineteen. That is often fine, but it can collapse the paragraph breaks in a document you meant to keep readable. If your input is a list of items, blank lines are usually noise, so it can help to strip them first. If your input is prose with paragraphs, deduplicating line by line is usually the wrong tool, because repeated lines such as "Thanks," or "Regards," may be legitimate.

Keep the original order or sort?

You can remove duplicates in two ways, and they give different output.

Keep first occurrence, preserve order. Walk through the list, remember each line you have seen and skip repeats. The first copy stays where it was. This is the right choice when order carries meaning: a ranked list, a log in time order, steps in a procedure, a mailing list you built in priority order.

Sort, then remove. Sorting puts identical lines next to each other, so duplicates collapse easily. The result is alphabetical. This is fine for tag lists, keyword lists and anything you will sort anyway.

The Unix Stack Exchange answer on this topic makes the distinction clearly: awk '!seen[$0]++' keeps the first occurrence of each exact line in the original order, while sort -u removes duplicates but sorts the output, as described in this Unix Stack Exchange thread.

A quick rule: if you cannot say why the order matters, you probably do not need to preserve it. If you can, keep it.

Comparison of order-preserving and sorted deduplication

Method 1: an online line remover in your browser

For most people with a pasted list, a browser tool is the fastest route. The usual workflow is:

  1. Paste your text into the input box.
  2. Choose the options: case-sensitive or not, trim spaces or not, sort or keep order.
  3. Run the tool and copy the result.
  4. Check the count of lines removed against what you expected.

The one thing worth checking before you paste anything private is where the processing happens. Many online tools send your text to a server. If your list holds customer emails, internal IDs or draft copy, prefer a tool that works locally in your browser so the text never leaves your machine. That is how every Fixora tool is built: files and text are processed in the browser, with no account and no upload.

Method 2: a text editor

If the text is already in a file, your editor may do the job without any extra tools.

Notepad++ (Windows). From version 8.1 onward there is a built-in command, found under Edit, then Line Operations, then Remove Duplicate Lines. You do not need a plugin, you do not need to sort first and you do not need regex, according to this StackOverflow answer. Notepad++ also has a separate command that removes only consecutive duplicates, as this Notepad++ walkthrough notes. Pick the one that matches your intent. The consecutive version leaves a repeated line alone if something else sits between the two copies.

Other editors. Most code editors have a "sort lines" command and many have a "delete duplicate lines" command, either built in or as an extension. Look in the command palette and search for "duplicate".

Method 3: the command line

On macOS, Linux or Windows with a Unix shell, three commands cover almost every case.

Keep first occurrence, original order

awk '!seen[$0]++' input.txt > output.txt

$0 is the full current line and seen is a record of lines already encountered. A line prints only the first time it appears. The explanation is the same one given in the Unix Stack Exchange thread above.

Sort and remove

sort -u input.txt > output.txt

Shorter, but the output is sorted.

The uniq trap

uniq input.txt

This is the command people reach for first, and it is the one that surprises them. uniq only compares adjacent lines. If the same line appears on line 2 and line 40, uniq keeps both. The POSIX specification states that repeated lines that are not adjacent are not detected, per the official uniq documentation. That is why you will often see sort input.txt | uniq: sorting first makes duplicates adjacent.

Method 4: a few lines of Python

If you want full control, such as case-insensitive matching with the original casing preserved, a short script is easy to adapt. The pattern is to track lines you have already seen in a set and write a line only when it is new. A StackOverflow answer shows this approach and points out that, unlike sort plus uniq, the script does not reorder lines, as this StackOverflow answer explains.

To make it case-insensitive and whitespace-tolerant, compare a normalized key (lowercase, stripped) but write out the original line. That way the first spelling you encountered survives.

Method 5: regular expressions

A find-and-replace with a backreference can delete consecutive duplicates in editors that support it. The pattern ^(.*)(\r? \1)+$, replaced with \1, works like this: the caret anchors at the start of a line, .* captures the whole line, the backreference \1 matches the same text again and the plus sign allows several repeats. This breakdown comes from a StackOverflow discussion of regex deduplication.

The catch is in the name: it only catches duplicates that sit next to each other. For a whole file you would sort first. With large files, regex backreferences can also be slow. If your editor has a dedicated command, use that.

Removing duplicates in a CSV or spreadsheet

This is where "remove duplicate lines" quietly becomes a different problem. A CSV row has several fields. Two rows might share an email address but differ in the signup date. Are they duplicates?

Only you can say. The sound approach is to choose the columns that define identity, instead of comparing the entire row. In Google Sheets, the route is Data, then Data cleanup, then Remove duplicates. You mark whether the data has a header row and untick the columns that should not take part. Rows that match on the selected columns count as duplicates even if the excluded columns differ, as the Sheets walkthrough describes.

There is a second decision hiding in there: which row survives. The UK Office for National Statistics guidance on duplicates makes the point that dropping duplicates across all columns is different from deduplicating on a subset, and that when you use a subset the surviving row can be arbitrary unless you set an explicit ordering rule, as covered in the ONS Spark guidance. In plain terms: if you have two rows for the same customer, decide whether you want the newest, the oldest or the most complete, and sort your data that way before you dedupe.

A practical order of work for a CSV:

  1. Keep a copy of the raw export.
  2. Tidy the file so that comparison is fair: consistent headers, trimmed spaces, a single delimiter. The CSV cleaner prepares files for import in your browser.
  3. If you only need certain columns to define a row, trim the file down with the CSV column selector.
  4. Remove the duplicates using the key you chose.
  5. If you need to see which rows changed between two versions, the CSV comparison by key shows differences row by row.

If you are comparing spreadsheet exports more broadly, our guide on comparing Excel files walks through finding every difference.

Check the result before you use it

Deduplication is destructive. Once the repeats are gone and you have saved over the file, you cannot tell what was removed. A few habits make it safe:

  • Keep the original. Work on a copy, always.
  • Check the counts. If you started with 1,200 lines and the tool says it removed 150, you should now have 1,050. If the numbers do not add up, something else changed.
  • Look at the diff. Put the original and the cleaned version side by side and scan only the differences. A text comparison tool highlights exactly which lines disappeared, so you can confirm that only true repeats were removed.
  • Spot-check the important items. For lists of names, figures, dates or URLs, read a handful of lines near the places where removals happened.

This audit-first idea is the core of safe text cleanup: do not trust any automated change until you have compared the output with the source.

Common problems and quick fixes

"The tool says no duplicates, but I can see them." Almost always whitespace or case. Turn on trimming and case-insensitive comparison, or clean the text first. Windows line endings (a carriage return before each line break) can also make the last line of a file look different from the rest.

"It removed lines I wanted to keep." You probably had legitimate repeats, such as a repeated header, a separator line or a recurring sign-off. Either dedupe a section at a time or restore from your original and use a key-based method.

"My list lost its order." You used a sorting method. Rerun with a keep-first-occurrence method.

"Only some duplicates were removed." You used a consecutive-only method such as plain uniq, or Notepad++'s consecutive command. Sort first or switch to a whole-file method.

"The file is huge." Browser tools handle typical lists easily. For files with millions of lines, the command line is usually the right place, because awk and sort stream through data without displaying it.

Our perspective

Most people who search for this have a pasted list and thirty seconds. They do not want to learn awk, install an editor or create an account to clean it.

We build Fixora around that moment. Every tool runs in your browser, and your text or files are processed locally, with no upload and no sign-up. For a list of emails, a draft or a client export, that matters more than any extra feature: the content stays on your device.

We also think text cleanup belongs in a chain, not in a single click. Tidy the input, remove what repeats, then compare the before and after. That is why our text, CSV and JSON tools sit side by side: you can clean a file, compare it with the original and move on without switching between several sites.

The advice we give ourselves is simple: decide what "the same" means before you delete anything, and keep your original until the result has been checked.

IA

Clean up text and data in your browser

If your duplicates are inside a CSV, our browser tools handle the surrounding work: tidying a file for import, choosing the columns that matter, comparing two exports and converting between CSV and JSON. Nothing is uploaded and no account is needed.

Fixorahub

Browse the full set of free tools on the Fixora tools page, or open the CSV and JSON workflow to clean, compare and convert a data file in one place.

FAQ

How do I remove duplicate lines but keep the first one?

Use a method that keeps the first occurrence and preserves order. In a shell, awk '!seen[$0]++' input.txt > output.txt does it. In a browser tool, choose the option that keeps order and leave sorting off. Avoid plain sort -u if the order matters, because it alphabetizes the result.

Why does uniq not remove all my duplicates?

uniq only compares neighboring lines, so repeats separated by other lines survive. Sort the file first with sort input.txt | uniq, or use sort -u or the awk one-liner instead. The POSIX documentation for uniq confirms that non-adjacent repeats are not detected.

Is duplicate-line removal case-sensitive?

By default, usually yes: Apple and apple are different lines. Many tools have an ignore-case option. If yours does not, convert everything to one case first, remove the duplicates, and keep your original text if the original capitalization matters.

How do I remove duplicates from a CSV without losing different rows?

Do not compare entire rows. Choose the columns that identify a record, such as an email address or ID, and remove duplicates on those columns only. Decide in advance which row should survive, for example the newest, and sort the data that way first. Then compare the result with your original to confirm nothing important was lost.

Is it safe to paste private lists into an online tool?

It depends on whether the tool processes text locally or sends it to a server. For emails, customer data or unpublished drafts, use a tool that runs in your browser without uploading. Fixora tools work that way, with no account required.

Sources

Made with BabyLoveGrowth to earn more organic traffic