How to Remove Duplicate Lines From a File Without Sorting

Deduplicating a list is easy when the order does not matter and surprisingly irritating when it does: a log excerpt, a list of URLs in priority order, a PATH with repeats. The standard one-liner is short enough to memorise, and understanding why it works lets you adapt it. It builds on sort and uniq, and the examples assume GNU awk 5.2 and coreutils 9.x.

Keep the first occurrence, keep the order

printf '%s\n' b a b c a d | awk '!seen[$0]++'

b
a
c
d

The expression seen[$0]++ returns the current count and then increments it. The first time a line appears the count is 0, which is false, so ! makes it true and awk prints the line (the default action of a pattern without braces). On later appearances the count is positive and the line is skipped. One pass, original order, no sorting.

The alternatives

sort -u file                       # sorted, unique
sort file | uniq                   # same result, two processes
sort -u -k2,2 file                 # unique by field 2, sorted
sort -s -u -k2,2 file              # stable: keeps the first line of each key
cat -n file | sort -uk2 | sort -n | cut -f2-   # classic order-preserving trick

sort -u is the right tool when order is irrelevant: it spills to disk for huge files and never runs out of memory. The numbering trick restores order for shells without awk but is clumsy. Without a prior sort, uniq removes only runs of identical adjacent lines, which is what you want for a log with repeated messages in a row, not for a global dedupe.

By field, by case, by trimmed text

awk '!seen[$2]++' data.txt                      # first row for each value in field 2
awk -F, '!seen[$1","$3]++' data.csv             # key from two columns
awk '!seen[tolower($0)]++' emails.txt           # ignore case
awk '{ k = $0; gsub(/^[ \t]+|[ \t]+$/, "", k) } !seen[k]++' list.txt

Any expression works as the key. Combine columns with a separator that cannot appear in the data. To keep the last occurrence instead of the first, reverse the file: tac file | awk '!seen[$0]++' | tac; see rev and tac. Deduplicating email addresses is a good example of normalising before comparing, since [email protected] and [email protected] are usually the same.

Edit a file safely

awk '!seen[$0]++' list.txt > list.txt.tmp && mv list.txt.tmp list.txt
gawk -i inplace '!seen[$0]++' list.txt

Never redirect output to the file you are reading: the shell truncates it before awk starts, and you lose the data. Write to a temporary file and move it over the original, as in editing in place on macOS and Linux. gawk's inplace extension does this for you, but it is not available in every awk.

Remove duplicates in PATH

export PATH=$(printf '%s' "$PATH" | awk -v RS=: '!seen[$0]++ { printf "%s%s", sep, $0; sep=":" }')

Setting the record separator to a colon turns the path into records. Printing with a separator that begins empty avoids a leading or trailing colon. Repeated entries come from sourcing profile files more than once.

Why the array grows and what that costs

The one-liner stores every distinct line as a key in an associative array, and awk keeps that array in memory until it exits. For a few hundred thousand short lines this is invisible. For a file with tens of millions of distinct lines, each key carries the string itself plus hash-table overhead, so the process can use several times the size of the data on disk. The symptom is a script that runs fast on a sample and then stalls on the real file as the machine starts swapping. Check the memory use of a run on a representative slice before pointing it at the full input, and extrapolate from the number of distinct lines, not the total.

There is a cheaper variant when only a hash of the line matters. Keying on a checksum or on a single column shrinks the stored string, at the price of a collision risk that is negligible for a 64-bit hash and real for a short prefix. Keying on the first 40 characters of a long line is a decision about what "duplicate" means, and it should be a deliberate one, since two different log lines that share a prefix would be collapsed into one.

Duplicates that are not byte-identical

Most real duplicates differ in invisible ways. A file edited on Windows has carriage returns at the end of some lines, so report and report\r look the same on screen and compare as different keys. A trailing space, a tab instead of spaces and a different Unicode normalisation of the same accented letter all produce lines that a person calls identical and awk does not. When the output still shows apparent duplicates, run the file through cat -A to reveal carriage returns and tabs, then normalise in the key expression rather than in the data: !seen[gensub(/\r$/, "", 1)]++ in gawk strips the carriage return for the comparison while printing the original line.

Case is the other frequent trap. tolower($0) works for ASCII and for the current locale in gawk, but a locale mismatch between the machine that wrote the file and the one reading it changes what counts as the same letter. If the data contains non-ASCII names, set the locale explicitly and test with one known pair, such as an uppercase and lowercase accented letter, before trusting the result on the whole file.

Keeping the last occurrence, or counting instead of dropping

Sometimes the newest entry is the one that matters, for example a configuration file where later assignments override earlier ones. Reversing the file around the filter works, and a two-pass awk script that records the last line number for each key and prints on the second pass avoids the double reversal and keeps memory use the same. Which occurrence to keep is a question about the data, and the answer should be written down next to the command, since the two choices produce different files from the same input.

When the goal is to understand the duplicates rather than remove them, count them. sort file | uniq -c | sort -rn lists each distinct line with its frequency, most common first, and the top few rows usually explain where the repeats come from: a retry loop, a cron job that ran twice, a profile file sourced from two places. Removing duplicates before knowing why they exist hides the bug that produced them, so look at the counts first and delete second.

Checking the result before replacing the file

Deduplication is destructive when it writes back over the source, so verify first. Compare the line counts before and after with wc -l, and confirm that the number of lines removed equals the number of repeats you expected. A mismatch usually means the key differs from the one you assumed, for example a trailing space in half the lines. Then check that the set of distinct lines is unchanged: sort -u original and sort -u result should be identical, and diff on the two reports nothing if the filter dropped only repeats. This takes a few seconds and catches the case where a typo in the key expression removed distinct data.

Keep the original until the downstream step has succeeded. Renaming the input to a backup name, or writing the result to a new file and switching a symlink, makes the change reversible. For files under version control the diff is the check, and for log files that are rotated anyway, deduplicating a copy is nearly always the better choice than touching the live file.

Where this does not apply

The associative array holds every distinct line, so a file with a hundred million unique lines needs gigabytes of memory; use sort -u there, which spills to disk. For near-duplicates, such as lines differing by a timestamp, normalise the key first by cutting the varying part. Binary data with NUL bytes needs -z in sort and different tools. See the gawk manual's uniq program and the sort manual.