How to Compare Bash Awk Python and Go on the Same Task

"Which is faster?" has one honest answer: it depends on the task, the data and how you write each version, so measure. A worked example helps show where the differences come from. The task here is counting requests per status code in an access log, implemented four ways, plus the classic pipeline. The method follows trustworthy benchmarks, and the example code assumes Bash 5.2, gawk 5.2, Python 3.13 and Go 1.23. Timing numbers are described qualitatively because they depend on hardware.

The task and the four versions

# Bash
declare -A c
while IFS=' ' read -r _ _ _ _ _ _ _ _ status _; do
  (( c[$status]++ ))
done < access.log
for k in "${!c[@]}"; do echo "$k ${c[$k]}"; done | sort

# awk
awk '{ c[$9]++ } END { for (k in c) print k, c[k] }' access.log | sort

# Python
import collections, sys
c = collections.Counter(line.split()[8] for line in sys.stdin if len(line.split()) > 8)
for k, v in sorted(c.items()):
    print(k, v)

package main

import (
	"bufio"
	"fmt"
	"os"
	"sort"
	"strings"
)

func main() {
	counts := map[string]int{}
	sc := bufio.NewScanner(os.Stdin)
	for sc.Scan() {
		f := strings.Fields(sc.Text())
		if len(f) > 8 {
			counts[f[8]]++
		}
	}
	keys := make([]string, 0, len(counts))
	for k := range counts {
		keys = append(keys, k)
	}
	sort.Strings(keys)
	for _, k := range keys {
		fmt.Println(k, counts[k])
	}
}

All four read the log once and count a field. Check they print identical output before timing: diff <(bash count.sh) <(awk ...). A benchmark of programs that compute different things is meaningless. The pipeline alternative is cut -d' ' -f9 access.log | sort | uniq -c.

Time them fairly

go build -o count-go count.go
hyperfine --warmup 2 --runs 10 \
  -n bash 'bash count.sh < access.log' \
  -n awk  "awk '{c[\$9]++} END {for (k in c) print k, c[k]}' access.log" \
  -n python 'python3 count.py < access.log' \
  -n go './count-go < access.log' \
  -n pipeline "cut -d' ' -f9 access.log | sort | uniq -c"

Use the same input file for all, warm the page cache with warmup runs (otherwise the first run pays for disk reads, as in the page cache), compile Go before timing, and use the same machine in one session. Run with several input sizes: at 1,000 lines process start-up decides; at 10 million lines the per-line cost does. Report mean and spread, and keep the raw output.

What you will usually see

Rank by speed on a large log (typical, not measured here):
  1. Go            compiled, tight loop
  2. awk / pipeline  C implementation, single pass or sort-based
  3. Python        interpreted, but the loop is simple and C-backed dict/Counter
  4. Bash          per-line interpreter overhead; slower by an order of magnitude or more

The ranking is typical and not a promise; GNU awk, mawk and Python versions differ, and mawk is noticeably faster than gawk on simple scripts. A Bash loop is slow because every line passes through the interpreter's parsing and variable machinery, and every external command inside it adds a fork and exec. See the cost of fork and exec. Python's speed comes from C code behind split and Counter; rewriting the same logic with explicit Python loops slows it. Go's advantage grows with input size and with CPU-heavy work.

Why the ratios change

Switching the task to "extract a regex group" favours tools with a good regex engine, and "join two files" favours sort-based Unix tools or a hash-based approach in a language. IO-bound tasks (reading from slow disk or network) erase the differences. Parallel Go or xargs -P changes everything again. Locale also matters: a UTF-8 locale makes sort and some awk functions much slower than LC_ALL=C, as in the LC_ALL=C speed-up.

Choosing from the result

Speed is one input. A script that runs once a day on a small file should be written in the language the team reads fastest. Pick Bash for glue under about 50 lines, awk or the pipeline for text summaries, Python when logic or libraries matter and Go when the tool is long-lived or heavy. See Go versus Bash for the broader trade-offs.

Why comparing languages is harder than it looks

A benchmark between languages measures a particular program, not the languages. The Bash version above spends its time in the interpreter's loop, the awk version in a tight C routine, the Python version in a mix of interpreter and C-implemented dictionary operations, and the Go version in compiled code. Change the task slightly, and the order can change. Counting a field is dominated by splitting lines and updating a table. Matching a complicated regular expression is dominated by the regex engine, and a hand-written Go loop may lose to awk if the awk build uses an optimised engine and the Go code uses the standard library's slower, linear-time one. Quote the task, the data and the versions together with any number.

Skill matters as much as the tool. An experienced Python programmer writes the counting loop with a Counter and a generator, while a naive version appends to lists and sorts at the end, and the two differ by a large factor. When comparing, either give each language an equally careful implementation or say plainly that you measured typical code, not best possible code.

Startup cost and the size of the job

Every runtime has a fixed cost before the first line runs. Awk and a compiled Go binary start in a millisecond or two. Python needs tens of milliseconds to initialise the interpreter and import modules, and a larger import set raises that. Bash starts quickly but pays for every external command it launches. For a file of a thousand lines, the processing time is smaller than the startup time for every candidate, and the fastest tool is the one that starts fastest. For a file of ten million lines, startup is invisible and throughput decides. Plot time against input size to see where the lines cross, which tells you at what size the choice begins to matter.

This also affects how you structure a script. A Python program invoked once per file in a loop pays its startup cost once per file; the same program given all the files at once pays it once. Moving the loop inside the program is a bigger improvement than any micro-optimisation of the loop body.

Memory and streaming

Throughput is not the only axis. The awk and Go versions stream the file and keep only the table of counts, so memory stays flat as the input grows. A version that reads the whole file into a list first, which is natural in Python with readlines(), uses memory proportional to the file and may begin swapping on a large log, after which its speed collapses. Check peak memory with /usr/bin/time -v and look at the maximum resident set size alongside the time. A fast program that needs eight gigabytes is not an improvement on a slower one that needs eight megabytes, if the machine has four.

Reporting results honestly

Publish the method with the numbers: the hardware, the tool versions, the input size and how it was generated, the exact commands and how many runs were taken. Give the mean and the spread, not a single run. Say what the comparison does not cover, such as parallel versions, other regex engines or different locale settings. When a result is surprising, check the output of every version for equality, rerun on a second machine and read the profile before telling anyone. A surprising result is more often a measurement error than a discovery, and the habit of checking is what separates a benchmark from an anecdote.

Where this does not apply

This benchmark measures a single-threaded text task. Network-heavy, database-heavy or startup-sensitive workloads behave differently, and the right answer there may be to avoid doing the work at all. Optimising before measuring is the subject of when not to optimise. See the hyperfine repository and the gawk performance notes.