How to Use awk Associative Arrays for Counting and Grouping

Counting how many times something appears is the job that sends people from sort | uniq -c to awk, because awk does it in one pass without sorting first. The enabling feature is the associative array: an array indexed by strings. Once you have it, grouping, deduplication and lookups all take a line or two. This post builds on top values with sort and uniq. Examples use GNU awk 5.2 and a web access log.

Count and rank

awk '{ hits[$1]++ } END { for (ip in hits) print hits[ip], ip }' access.log | sort -rn | head -n 3

1842 203.0.113.9
911 198.51.100.4
377 192.0.2.77

The first field of the log is the client address. hits[$1]++ reads as: look up the entry for this address, treating a missing one as zero, and add one. At the end the loop prints count then key, and sort -rn ranks them. Doing the same with sort | uniq -c | sort -rn sorts all lines first, which is slower on large logs.

Sum by key and track more than one thing

awk '{ bytes[$1] += $10; reqs[$1]++ }
  END { for (ip in bytes) printf "%s %d %d\n", ip, reqs[ip], bytes[ip] }' access.log | sort -k3,3nr | head -n 3

Two arrays with the same keys keep two statistics. Field 10 of the combined log format is the response size. Add a third array for the maximum, or a status-code breakdown with a composite key, as below.

Composite keys

awk '{ n[$1, $9]++ } END { for (k in n) { split(k, p, SUBSEP); print p[1], p[2], n[k] } }' access.log | sort | head -n 4

The expression n[$1, $9] joins the address and the status code with the special character SUBSEP (by default \034), and split takes it apart again. Using a real separator in a string key, n[$1 "|" $9], is more readable and works the same, as long as the separator cannot appear in the data. gawk also offers true multidimensional arrays (arr[a][b]), which are not portable.

Test membership without side effects

awk 'NR == FNR { seen[$1]; next } !($1 in seen)' blocked.txt access.log

The statement seen[$1] by itself creates the key. The in operator tests for a key without creating it, whereas reading seen[$1] in a condition creates an empty entry as a side effect and can make later in tests true. The example loads a block list and prints log lines whose address is not in it; the technique with NR == FNR is covered in two files with NR and FNR.

Remove duplicates, keep order

awk '!seen[$0]++' urls.txt

The first time a line appears, seen[$0] is zero, so !0 is true and the line prints; the increment then makes it nonzero. Later copies are skipped. This preserves the original order, unlike sort -u. Memory grows with the number of distinct lines, so for a file with hundreds of millions of unique lines, a sort-based approach is the safer one.

Deleting and sizes

delete arr[k] removes one entry and delete arr removes all of them. length(arr) returns the number of entries in gawk, mawk and the one-true-awk, and is part of POSIX since the 2024 edition. Older awks accept it only as an extension, so count with a loop if you need strict portability.

Where this does not apply

Order of keys is undefined and differs between awk implementations, so never rely on it. For huge cardinality, a database or sort handles data that does not fit in memory. When the keys need numeric ordering, sort afterward with -n; gawk's PROCINFO["sorted_in"] is a convenience only on gawk. The commands for fields still apply.

How arrays work and a trap that creates elements

Awk arrays are associative: the index is a string, even when it looks like a number, and an array grows as you use it. The trap is that merely reading an element creates it. A test like if (count[$1] == 0) adds an element with an empty value for every key it checks, which changes the array's size and the output of a later loop. To ask whether a key exists without creating it, use ($1 in count). This matters for deduplication, set membership and any logic that later counts or prints the array. A related point is that a never-set element compares equal to both zero and the empty string, which is convenient for counters that start at zero and wrong for logic that needs to tell "never set" from "set to zero". Use the in operator when the difference matters.

Deleting an element uses delete arr[key], and delete arr with no index clears the array in all current implementations. The number of elements is available with length(arr) in gawk, mawk and the one true awk.

Keys that combine fields and the order of output

A composite key is made by joining fields into one string. The comma form arr[$1, $2] uses the built-in separator SUBSEP, a character that rarely appears in text, and it is checked with (($1, $2) in arr). Splitting a composite key back into parts uses split(key, parts, SUBSEP). Concatenating with an ordinary character such as a colon works too, and collides when the data contains that character, so a:b plus c and a plus b:c look the same. The order in which a for (k in arr) loop visits keys is implementation-defined and can differ between awk versions and even between runs, so a script that depends on it is broken in a way that shows up later. Pipe to sort for stable order, using sort -k2,2nr to rank by count, or use gawk's sorting facilities.

The one-liner !seen[$0]++ prints each distinct line the first time and keeps the original order, which sort -u does not. It works because the post-increment returns the old value, zero the first time, which is false, and the negation makes the rule true. The cost is memory proportional to the number of distinct lines.

Practical limits

An array holding every distinct key can become large on big inputs, and the process is killed when memory runs out, with a message that is easy to miss in a pipeline. Check the cardinality first with a cheap estimate, or sample the data. When the keys are sorted already, a streaming approach that keeps only the current group in memory is far cheaper: compare the key with the previous one, print the total when it changes, and reset. For very large data, sort and uniq -c use disk and cope with sizes that awk cannot, at the cost of speed. For repeated analysis, load the data into a database and ask questions in SQL. Awk is the right tool for one-off summaries and for pipelines where each pass is simple and the result is small.

See the gawk arrays chapter and POSIX awk arrays.