How to Count Unique Values and Top Items With sort uniq

"Which IP hits us most?" "Which error is most common?" "Which word appears most?" Each is a frequency table, and the shell idiom for it has been the same for decades. It is worth knowing exactly why each stage is there so that variations come naturally. It uses sort and uniq and cut, and the examples assume GNU coreutils 9.x.

The recipe

cut -d' ' -f1 access.log | sort | uniq -c | sort -rn | head -5

   4210 203.0.113.7
   1893 198.51.100.23
    977 192.0.2.44
    540 203.0.113.99
    301 198.51.100.8

Each stage has one job. cut extracts the first field, sort groups equal lines next to each other, uniq -c collapses runs and prefixes the count, sort -rn orders numerically from largest, and head keeps the top five. The addresses are documentation examples. Skip the first sort and the counts are wrong, because uniq sees the same address in separate runs.

Variations

sort file | uniq -d          # lines that appear more than once
sort file | uniq -u          # lines that appear exactly once
sort -u file                 # unique lines, no counts
tr -cs 'A-Za-z' '\n' < text.txt | tr 'A-Z' 'a-z' | sort | uniq -c | sort -rn | head -10

The last line is word frequency: tr -cs replaces every run of non-letters with a newline, so each word gets its own line, and the second tr folds case. uniq -c output has leading spaces, so reshape it with awk '{print $2, $1}' if you need the value first. Add -k2 to sort if you need to sort by a later field, as in sorting by column.

The awk alternative

awk '{ c[$1]++ } END { for (k in c) print c[k], k }' access.log | sort -rn | head -5

Awk counts in one pass over the input and only sorts the distinct keys at the end. For a log with 50 million lines and 40,000 distinct addresses that sorts 40,000 lines instead of 50 million. The difference can be a factor of ten in time and memory. If the file is huge, set LC_ALL=C to make sorting bytewise, as in speeding up with LC_ALL=C.

Ties and stability

sort | uniq -c | sort -k1,1nr -k2,2 | head

When counts tie, sort -rn breaks ties by comparing the entire line in reverse, which can scramble alphabetical order. Specify a key for the count and a second for the value to get a stable, predictable ranking. Report the total too: awk '{s+=$1} END {print s}' over the counts tells you what share the top ten represent.

Why sort comes before uniq

The uniq command collapses only adjacent duplicate lines. It keeps no table of what it has seen, which is why it uses almost no memory and works on endless streams, and why the input must be sorted first for the count to be correct. Running uniq -c on unsorted data reports each run of identical neighbours separately, so a value that appears at the start and the end of a file is counted twice, as two entries with small counts. The mistake is quiet because the output looks plausible. Always put sort before uniq, or use sort -u when only the distinct values matter and no count is needed.

The second sort, sort -rn, orders by the count. The -n flag matters because the counts are numbers, and without it the string "9" sorts after "10" and the top of the list is wrong.

Performance on large inputs

Sorting dominates the cost. GNU sort spills to temporary files when the input exceeds its memory buffer, which is slow on a spinning disk and tolerable on SSD. Three settings help: LC_ALL=C switches to byte comparison, often several times faster than locale-aware collation; --parallel uses several cores; and -S raises the buffer size, such as -S 2G, if the machine has memory to spare. Set -T to a fast directory with enough space for temporary files when the default location is small, since a full /tmp makes sort fail midway on a large file.

If the number of distinct values is small compared with the number of lines, an awk script with an associative array does the counting in a single pass with no sort at all, and then only the short result needs sorting. That scales to files far larger than memory-bound sorting would handle, as long as the distinct keys fit in memory.

Counting what you meant to count

The result is only as good as the key extraction before it. Extract the field with cut or awk and normalise it: lower-case with tr when case should not matter, strip surrounding whitespace, and remove port numbers or query strings when counting hosts or paths. A web log has the client address in the first field, but a proxy in front of the server may put every request under one address, and the real client is in a forwarded header. Check what the field holds before trusting the ranking. Likewise, a status code count that includes health checks from a load balancer will show those checks at the top, and the interesting traffic needs filtering out with grep -v first.

Print percentages to put counts in context. A small awk step after the sort computes each count divided by the total, which tells the reader whether the top entry is a majority or a sliver.

Presenting and keeping the result

Limit the output with head, and keep the full list in a file when someone might ask for item twenty-one. Right-align the counts with awk '{printf "%8d %s\n", $1, $2}' so a column of numbers is easy to compare. When the report is regular, write it as a script with the date and the input file name in the header and run it from cron. Keep the previous output and diff against it, because the change in the ranking is often more informative than the ranking itself. A new entry that jumps to the top on a Tuesday is a lead worth following, and it is invisible if each report is read alone.

Test the pipeline with a tiny known input before pointing it at a large file: five lines with two repeated values and one tie. Confirm the counts, the order and the handling of the tie, then scale up. Ties are ordered by the byte order of the line itself when sort -rn falls back to its last-resort comparison, which is stable enough to rely on only if the locale is fixed. Set LC_ALL=C in the script so the output is the same on every machine that runs it.

Finally, remember that a ranking hides the long tail. Report how many distinct values there were and what share the top ten cover, so a reader can tell a concentrated distribution from a flat one.

Where this does not apply

For counts by two dimensions, such as status by hour, build a composite key in awk. For data with a header and quoted fields, a CSV-aware tool or SQL GROUP BY is cleaner. Distinct counts at huge scale can use approximate counting, but a plain pipeline is exact up to memory. See the GNU uniq manual and the sort manual.