How to Write a Benchmark You Can Trust in Real Scripts

Most benchmark results in blog posts and pull requests cannot be repeated: one run, a laptop under load, a warm cache for one version and a cold one for the other. A reader acting on them may make things worse. A modest discipline makes your own numbers believable, and it is the same discipline used in regression tests with hyperfine and the pitfalls in why a benchmark lies. The examples assume Linux, hyperfine 1.19 and Bash 5.2.

The checklist

1. Same work      outputs compared and identical
2. Same input     one fixed file, checked in or generated with a fixed seed
3. Same machine   both versions, same session, alternating if possible
4. Warm up        discard the first runs
5. Repeat         10 to 30 runs, not one
6. Summarise      mean and standard deviation, or median and range
7. Control cache  all warm or all cold, stated
8. Record         versions, CPU, kernel, command, date

The list is short because each item prevents a specific kind of false result. Skipping the first is the commonest and most embarrassing: a "faster" version that produces less output.

Check the output first

cmp <(./old.sh data.csv) <(./new.sh data.csv) && echo "outputs identical"
sha256sum <(./old.sh data.csv) <(./new.sh data.csv)

outputs identical

A faster script that quietly skips edge cases is not faster. Compare the output byte for byte, or at least the checksum, on the benchmark input. Sort first if the order is not meant to be stable.

Measure with repetition

hyperfine --warmup 3 --runs 20 --export-json result.json \
  -n old './old.sh data.csv > /dev/null' \
  -n new './new.sh data.csv > /dev/null'

Benchmark 1: old
  Time (mean ± σ):     1.284 s ±  0.031 s    [User: 1.201 s, System: 0.071 s]
Benchmark 2: new
  Time (mean ± σ):     412.7 ms ±  9.2 ms    [User: 370.0 ms, System: 38.1 ms]

Summary
  new ran
    3.11 ± 0.10 times faster than old

The figures are an illustration of the report format. A 3x difference with a 2 percent spread is a real result; a 4 percent difference with a 10 percent spread is noise. Send the output to /dev/null so you measure the program and not the terminal. Where hyperfine is not available, a loop with time works: take the median of at least ten runs; see the time command.

Control the cache

hyperfine --warmup 1 'grep -c ERROR big.log'                           # warm cache
hyperfine --prepare 'sync; echo 3 | sudo tee /proc/sys/vm/drop_caches >/dev/null' 'grep -c ERROR big.log'   # cold cache

Say which one you measured. A warm-cache benchmark measures CPU and memory; a cold-cache one measures the disk. Real use is usually between. The details of how the cache distorts results are in the Linux page cache. For CPU-bound work, also note the CPU governor: cpupower frequency-set -g performance reduces drift on a desktop.

Reduce noise

uptime
sudo systemctl stop unattended-upgrades 2>/dev/null
taskset -c 2 ./new.sh data.csv > /dev/null
nice -n -5 ./new.sh data.csv > /dev/null

Check the load average before you start. Close browsers and indexers, and run on AC power. Pinning to one core with taskset removes scheduler variation but changes the workload; use it for single-threaded code. Virtual machines and shared CI runners add noise from neighbours, so repeat benchmarks across days before drawing conclusions, and compare ratios inside one run, not absolute times across runs.

Report it so others can repeat it

Hardware:  8-core x86-64, 16 GB RAM, NVMe
OS:        Debian 13, kernel 6.12
Tools:     bash 5.2, gawk 5.2, hyperfine 1.19
Input:     access.log, 2.1 million lines, 410 MB (checksum recorded)
Command:   hyperfine --warmup 3 --runs 20 ...
Result:    new is 3.1x faster (mean 413 ms vs 1.28 s), spread under 3 percent

A claim with a number and no setup cannot be tested. Include the input description and the command. If the result is surprising, repeat it on another machine; if it holds, you can trust it.

Why a single fast run proves nothing

A command's run time is a distribution, not a number. The first execution pays for loading the binary and its libraries from disk, later ones find them in the page cache. The CPU may be running at a low clock until load arrives, a background indexer may wake up halfway through, and on a laptop thermal throttling can slow the fifth run compared with the first. Reporting the best of three hides all of this, and reporting the first hides it differently. Only a set of runs, summarised with a spread, tells a reader how much of the difference is real.

The spread is as informative as the mean. A tool that takes 412 ms with a standard deviation of 9 ms is stable, and a 30 percent improvement over it is easy to see. A tool with a mean of 1.2 s and a deviation of 0.4 s is noisy, and a claimed 10 percent gain is inside that noise. If the deviation is large relative to the difference, the honest conclusion is "no measurable difference on this setup", and that is a fine thing to publish.

Comparing the right things

The most useful habit is to change one thing at a time. A benchmark that swaps both the algorithm and the language, or the tool and the input format, cannot say which change produced the gain. Keep the input, the machine, the flags and the shell constant, and vary the single variable under test. When the comparison is between two different tools, make sure both are asked the same question: a grep that stops at the first match and a script that reads the whole file are not doing the same work, even though they answer a similar question.

Input size deserves its own check. A tool that is slower on ten lines can be faster on ten million, because startup cost dominates small inputs and algorithmic cost dominates large ones. Run at two or three sizes and look at how the time grows. If doubling the input doubles the time, the cost is linear and a single size is representative. If it quadruples the time, the result at one size will not predict the result at another, and the claim needs the size in it.

Reading the result with suspicion

Before publishing a number, try to break it. Run the benchmark in the opposite order, new before old, and see if the winner changes; ordering effects from the cache and from CPU frequency are common, and alternating the two is the standard defence. Re-run on a second day, or on a second machine, and check that the ratio between the two versions holds even if the absolute times move. Ratios survive a change of hardware better than seconds do.

Also ask whether the gain matters. A script that drops from 400 ms to 200 ms is twice as fast, and for a job that runs once a night it is also irrelevant. State the absolute saving next to the ratio, and say how often the command runs, so a reader can decide whether the change is worth the added complexity. A benchmark exists to support a decision, and a number without the decision it supports is only decoration.

Keep the raw output. The JSON export from the benchmarking tool, the exact command line and the input checksum take a few kilobytes, and they let anyone, including you in six months, re-run the comparison and see whether a newer version of a tool changed the answer. Results from an old release are a record of that release, and the version numbers in the sentence make that clear.

Where this does not apply

These rules suit command-line tools and scripts. Services under concurrent load need load-testing methods with percentile latencies, warm-up of JIT and connection pools, and a steady-state period. Microbenchmarks of very short code need special tools, since the measurement overhead dominates. See the hyperfine documentation and Brendan Gregg's active benchmarking method.