How to Use awk to Parse Logs and Build Reports the Safe Way

Logs are text with structure, and awk is built for text with structure. A report is a loop that groups lines by a key and keeps a total, which in awk is one line. The language rewards learning four idioms, and the same idioms handle web server logs, application logs and CSV exports. They extend awk fields and printf, and the examples assume GNU awk 5.2 on a combined-format access log.

Count requests by status

awk '{ status[$9]++ } END { for (s in status) print s, status[s] }' access.log | sort -k2 -rn

200 18432
304 2210
404 511
500 37

In the common log format, field 9 is the status code. The array status is created on first use, so no initialisation is needed. The for (s in status) loop has no guaranteed order, which is why the output goes through sort. For a ranking, the same job done in plain shell is sort and uniq counting; awk wins when you need more than a count.

Sum and average by key

awk '$9 == 200 { bytes[$7] += $10; hits[$7]++ }
     END { for (p in bytes) printf "%-30s %8d hits %10.1f KB avg\n", p, hits[p], bytes[p]/hits[p]/1024 }' access.log \
  | sort -k2 -rn | head -5

/index.html                       9120       12.4 KB avg
/api/items                        4410        3.1 KB avg
/static/app.js                    2987      212.8 KB avg

The pattern $9 == 200 before the braces limits the action to matching lines. Two arrays keep a sum and a count, and the average is computed at the end. The figures are illustrative. Division by zero cannot occur because the key exists only if at least one line added to it.

Bucket by time

awk '{ split($4, t, ":"); hour[t[2]]++ } END { for (h in hour) print h, hour[h] }' access.log | sort
awk -F'"' '{ split($2, req, " "); method[req[1]]++ } END { for (m in method) print m, method[m] }' access.log

split breaks the timestamp [03/Nov/2026:09:41:02 on the colon, and the second element is the hour. Changing the delimiter with -F'"' gives access to the quoted request line as field 2, then split again on spaces. Quoted fields with embedded delimiters are the weakness of awk's field splitting; see awk and CSV.

Guard against bad lines

awk 'NF >= 10 && $9 ~ /^[0-9][0-9][0-9]$/ { ok[$9]++; next } { bad++ } END { print "bad lines:", bad+0 }' access.log

Check that the line has enough fields and that the status looks like a number before using it. Count the rest to see whether the parser, not the data, is wrong. bad+0 forces numeric output when the variable never got set, so you print 0 instead of an empty string. When regular expressions in awk are the problem, see awk regex matching.

Filter by time window

awk -v from="03/Nov/2026:09" '$4 ~ "^\\[" from { n++ } END { print n+0 }' access.log

Passing the pattern with -v avoids splicing shell variables into the program, which is safer; the reasoning is in passing shell variables into awk. Time filtering by string prefix works because the format sorts naturally within an hour.

Finding the format before writing code

Every log analysis begins by looking at real lines. Print a few with head and a few from the middle with sed -n '1000,1005p', and note the field separator, the position of the timestamp, how the request line is quoted and whether any fields can contain spaces. The common web server format has the client address, a timestamp in brackets, the quoted request, status and size. Splitting on spaces puts the quoted request across three fields, which is fine if you only need the first, the status and the size, as long as the fields before and after the quote are counted correctly. Alternatively, use a double quote as the field separator to extract the request in one piece.

Check how consistent the file is: count fields per line with awk '{print NF}' | sort | uniq -c. Lines with an unusual count point at malformed requests or a changed format, and the script should decide what to do with them.

Percentiles and averages

A mean of response times hides the slow tail. Report the median and the 95th or 99th percentile as well. Awk has no percentile function, but you can collect values in an array, sort them with gawk's asort and index into the sorted array. For POSIX awk, pipe the values to sort -n and pick the line at the right position with awk or sed. With a hundred thousand lines this is fast. State the units: a field in microseconds and one in milliseconds in the same log is a trap, and the first sanity check of any report is that the numbers have a plausible magnitude.

Count totals as well as ratios. An error rate of five percent on ten requests and on ten million requests mean different things, so print both numbers.

Comparing periods and spotting change

A single report tells you what happened, and a comparison tells you what changed. Run the same script on yesterday's and today's logs, and print the difference for each key. A script can read two files and keep two sets of counters, or you can generate two outputs and join them with join after sorting on the key. A new status code, a path whose count tripled and a host that disappeared are the findings worth the effort. Keep each day's summary in a dated file, so trends over weeks come from small files and not from re-reading huge logs. Rotated and compressed logs can be read with zcat piped into the same script, which makes the script's input a stream, independent of how the files are stored.

Take care with time zones. Web server logs carry an offset in the timestamp, and a report that groups by hour should say whether it is in server time or UTC.

Making the script a tool

When an awk report is useful once, it is usually useful again. Put the program in a file with a header comment, take the input from standard input or file arguments, and pass parameters with -v instead of editing the program. Write the expected output for a small sample log next to it and diff against it when you change the script. Handle the end cases: an empty input should print a clear message, not nothing. Decide the scale at which awk stops being the right tool: at hundreds of gigabytes, or when you need joins with other data, a database or a log platform is better. Until then, a small, tested awk script that answers a recurring question in seconds is hard to beat for cost and clarity.

Finish a report with the evidence a reader needs to trust it: the log files and period covered, the number of lines read, the number skipped as malformed and the time the report ran. Those few lines answer the first question anyone asks about a number, which is where it came from. Store the script and its output together, with the date, so a figure quoted in a meeting can be traced back to the exact input a month later.

Where this does not apply

For one-off counts, cut | sort | uniq -c is faster to write. For gigabytes of logs with repeated analysis, load them into SQLite or use a log tool. Different log formats shift field numbers, so print NF and a sample line when a report looks wrong. See the GNU awk manual and the POSIX awk specification.