How to Use hyperfine for Performance Regression Checks

Performance degrades quietly: each change adds a few milliseconds until a script that took two seconds takes ten. A benchmark that runs on every change turns that into a failing test. The tool for command line programs is hyperfine, and the main difficulty is not running it but choosing a fair comparison and a threshold that does not cry wolf. It builds on the time command and writing a benchmark you can trust. The examples assume hyperfine 1.19.

Measure properly

hyperfine --warmup 3 --runs 20 './process.sh data/large.csv'

Benchmark 1: ./process.sh data/large.csv
  Time (mean ± σ):     412.6 ms ±  9.8 ms    [User: 380.1 ms, System: 29.4 ms]
  Range (min … max):   399.8 ms … 437.2 ms    20 runs

The numbers shown are an example of the output format. --warmup runs the command a few times first so that file caches and CPU frequency settle, and --runs sets the sample size. The spread matters as much as the mean: if the standard deviation is 20 percent of the mean, a 5 percent regression is invisible. hyperfine also warns about outliers and about commands that run faster than a few milliseconds, where the shell start-up dominates.

Compare old and new

git stash -q 2>/dev/null; git worktree add -q /tmp/base origin/main
hyperfine -w 3 -r 20 -n base '/tmp/base/process.sh data/large.csv' -n head './process.sh data/large.csv'

Summary
  base ran
    1.04 ± 0.03 times faster than head

Benchmarking both versions in one invocation, on the same machine and in the same minute, removes most environmental noise. Here the new version is about 4 percent slower, inside the spread, so no conclusion. Names with -n make the summary readable. A git worktree gives the baseline a separate checkout without disturbing your work; see branches for naming them.

Export and gate

hyperfine -w 3 -r 20 --export-json bench.json -n base '/tmp/base/process.sh data/large.csv' -n head './process.sh data/large.csv'
python3 - <<'EOF'
import json, sys
r = {x["command"].split("/")[0] or x["command"]: x for x in json.load(open("bench.json"))["results"]}
base, head = (x["mean"] for x in json.load(open("bench.json"))["results"])
ratio = head / base
print(f"head/base = {ratio:.3f}")
if ratio > 1.15:
    sys.exit(f"regression: {ratio:.0%} of baseline")
EOF

The JSON holds mean, stddev, min, max and per-run times for each command. The gate compares the mean of the new version to the baseline from the same job and fails above a 15 percent increase. A generous threshold avoids false failures on shared CI machines; tighten it only if the noise is low. Comparing ratios within one run is much more stable than comparing today's time with last week's stored number, which depends on the machine of the day.

Make the workload stable

hyperfine --prepare 'sync; echo 3 | sudo tee /proc/sys/vm/drop_caches >/dev/null' 'grep -c ERROR big.log'
hyperfine --shell=none 'sort -u words.txt'
hyperfine -L n 1000,10000,100000 'seq {n} | ./sum.sh'

--prepare runs a setup command before each timing, here dropping the page cache for a cold-cache measurement (needs root); without it, runs after the first are warm, as explained in the page cache. --shell=none avoids measuring the shell itself for simple commands. -L parameterises the input size, which shows how the cost scales, useful for catching an accidentally quadratic change. Use a fixed, checked-in dataset so the workload does not drift.

Where performance regressions come from

Few regressions arrive as one dramatic change. They accumulate: an extra subshell in a loop, a pipeline that now runs sort twice, a call to an external program that used to be a shell built-in, a dependency update that adds start-up work. Each costs a few milliseconds and passes review because it looks harmless. A benchmark in CI works by making the sum visible after every change instead of after twenty, when finding the guilty commit means bisecting through weeks of history.

The location of the cost is as useful as its size. When a gate fails, the next question is which part of the run got slower. Keep a second, finer benchmark for the stages of the pipeline, or run with strace -c or perf stat to see where the time goes. A change that adds a thousand process launches shows up as a jump in system calls long before the wall time is clearly worse, which is why counts are often better indicators than seconds.

Making the baseline trustworthy

A baseline that is wrong makes every comparison wrong. Measure the base and head versions in the same job, on the same runner, alternating if the tool allows it, so slow hardware affects both equally. Build both from clean checkouts with the same tool versions, since a different compiler or interpreter between the two is a change you did not intend to measure. Record the runner type, the operating system and the tool versions in the output file, and when results change abruptly without a code change, compare those details first.

Shared CI runners introduce noise that no threshold can remove. Neighbouring jobs compete for CPU and disk, so the spread varies from run to run. If a dedicated runner is not possible, take more samples, use the minimum or the median instead of the mean, and repeat a failed comparison once before reporting it. A regression that appears twice in a row is real. One that appears once is probably the neighbour.

Choosing the workload

The benchmark input decides what is measured. Pick a case that reflects real use: a log of realistic size, a repository with real history, a document with the structure users have. Check it in or generate it from a fixed seed, and note its checksum. Add a second, much larger input if the algorithm's scaling matters, because the cost of a quadratic loop is invisible at a hundred lines and decisive at a million. Keep the whole benchmark short, under a couple of minutes, so that it runs on every change; a suite that takes twenty minutes gets skipped.

Handling the failure

A failed performance gate should be easy to act on. Print the two means, the ratio and the threshold, and link the full JSON as an artifact. Allow an override with a label or a commit trailer for intentional trade-offs, such as a feature that costs time on purpose, and require a sentence explaining it. Review the threshold quarterly. If it never fails, it may be too loose to protect anything, and if it fails often for no reason, it trains people to ignore it, which defeats the point of having a gate.

Reading the numbers with the right statistics

A mean hides the shape of the data. Two commands with the same mean can behave very differently: one steady at 400 ms, the other alternating between 200 and 600. When a gate looks only at the mean, the second one passes and users feel the slow runs. Export the individual timings and compare the median and the maximum as well, and for commands whose time varies with disk state, the minimum is often the most stable indicator of the code's own cost because noise only adds time, never removes it.

Use a significance check when the difference is small. With twenty samples per side and similar spread, a difference smaller than twice the standard error is indistinguishable from noise. A rough rule in the gate, requiring both a ratio above the threshold and a gap larger than the combined spread, removes most false alarms without hiding real slowdowns, and the extra three lines of Python are cheaper than the time spent chasing phantom regressions.

Where this does not apply

Microbenchmarks of tiny commands are dominated by process start-up. Latency-sensitive services need load tests with percentiles, not a mean. Noisy shared runners may be unusable for tight thresholds; a dedicated runner, or counting instructions with perf stat, is steadier. CPU-frequency scaling and thermal throttling change results on laptops. See the hyperfine repository and perf stat.

The perf-stat manual page describes counting instructions and cycles, which is steadier than wall time on noisy machines, and is a useful second signal beside hyperfine when a regression is borderline.