Why a Benchmark Lies and How Caches Change Results

You run a script twice and the second run takes a tenth of the time. Did your optimisation work, or did the first run just load the file? Caches at every level speed up repetition, and a benchmark that does not account for them measures the cache. The remedy is to control the state on purpose. The mechanism is in the Linux page cache, and the method in trustworthy benchmarks. The examples assume Linux 6.x and util-linux 2.40.

See the cache effect

sync; echo 3 | sudo tee /proc/sys/vm/drop_caches >/dev/null
time grep -c ERROR big.log
time grep -c ERROR big.log

8421

real    0m4.812s
user    0m0.940s
sys     0m0.310s
8421

real    0m0.981s
user    0m0.930s
sys     0m0.050s

The same command takes almost five seconds when the file has to come from disk and about one second when the kernel already holds it in memory. The CPU time (user) is the same in both; the difference is waiting for the disk. Numbers here show the pattern, not measurements from a particular machine. A result like "the new version reads the file 5x faster" may be nothing but the second run.

Check whether a file is cached

fincore big.log
vmtouch big.log
cat /proc/meminfo | grep -E '^(Cached|Buffers|MemAvailable)'

RES  PAGES  SIZE  FILE
 2G   524k  2.1G  big.log

fincore (part of util-linux) shows how many pages of a file are resident in the page cache. A fully resident file means a warm run; a mostly empty one means cold. The kernel also decides on its own what to evict, so a benchmark done after other work may not be as warm as you think. The drop_caches trick needs root and affects the whole machine, including other users' performance, so do it on a test machine.

Choose the state and be consistent

# warm: reflects steady-state use of a hot file
hyperfine --warmup 3 'grep -c ERROR big.log'

# cold: reflects the first access after boot or eviction
hyperfine --runs 5 --prepare 'sync; echo 3 | sudo tee /proc/sys/vm/drop_caches >/dev/null' 'grep -c ERROR big.log'

Neither is "right". A log analysis tool run once a day sees a cold file; a build step running repeatedly sees warm ones. If the question is whether your code is efficient, measure warm to remove disk noise. If it is about the user's experience of the first run, measure cold. State your choice with the numbers, and never compare numbers from different states.

Other caches that lie

CPU caches / branch predictor   first loop iterations slower; warm-up runs fix it
DNS and TCP connection reuse    second request skips lookup and handshake
HTTP / CDN / reverse proxy      served without touching the application
Application / ORM caches        second query served from memory
Compiler and build caches       second build reuses object files
JIT warm-up (JVM, Node)         early iterations slower than steady state
Filesystem metadata (dentries)  find, ls and stat are fast on the second pass

Each shows as a first run that is slower than the rest. For API tests, add a cache-busting parameter or Cache-Control: no-cache if you want to measure the origin, and measure the cached path separately. For builds, run with a clean cache for the baseline and with a warm one for incremental behaviour, as in speeding up Docker builds.

Spot a lying benchmark

Signs: the first run is far slower than later ones; user plus sys time is much lower than real time (waiting on IO); results change after a reboot; a faster version is faster only when run second; results differ between machines by more than the hardware explains. When in doubt, alternate the order of the two versions and see whether the "winner" changes.

Cold, warm and what users actually experience

Both cold and warm numbers are legitimate, and the mistake is mixing them. A tool that people run after a long gap, like a monthly report, mostly meets cold caches. A command run repeatedly in a build loop mostly meets warm ones. Decide which describes real use, measure that state on purpose and say which it was in the report. If you cannot decide, measure both and show both, since the difference is itself information: a large gap says the tool is IO-bound, and a small gap says it is CPU-bound, which points to where optimisation effort will pay.

Cold measurements need a recipe that works every time. Dropping the page cache needs root and is a blunt instrument that also affects everything else on the machine, so run it on a test machine and not on a shared server. In a container or virtual machine, a fresh start is a more honest cold state than dropping caches inside a running system, because it also resets the file system metadata and the hypervisor's own cache.

Layers you may not control

Below the operating system sits more caching. Disks have internal caches, RAID controllers have battery-backed caches and cloud block storage has its own read-ahead and burst behaviour. A cloud volume may be fast for the first minutes of a test and throttled afterwards, because of credits that run out, so a short benchmark overstates sustained throughput. Run long enough to see the steady state, watch the device statistics with iostat -x 1 and note the instance and volume type in the report. Results from a laptop SSD, a cloud volume and a spinning disk are different measurements of different systems.

Network storage adds another layer. Files on NFS or a similar share have client-side caching governed by mount options, and a benchmark on a second client may find the first client's changes not yet visible. When measuring over a network, either disable client caching for the test or accept that you are measuring the cache policy.

Designing the test to resist caching

Where you want to measure the work itself, give each run fresh input. Generate a new data file per run, or rotate through several files larger than memory so earlier ones are evicted. Use unique query parameters for HTTP tests. For compilation, clean the build directory. Make the cost of preparation visible by timing it separately; it should not be inside the measured loop. A harness that reports the preparation time and the measured time side by side makes it obvious when the cache is doing the work.

Writing it down

State the cache condition in the first line of a benchmark report, next to the hardware and versions. "Warm page cache, file of 2.1 GB, 16 GB RAM" lets the reader judge whether the numbers apply to their case. If a result depends on caching, say so plainly, and avoid headlines such as "ten times faster" without the condition. Honest conditions make claims narrower and trusted, which is what a measurement is for.

A short checklist keeps you honest. Name the cache state. Run each variant the same number of times in alternating order. Report the spread. Re-run the best and worst cases once more on a different day. If the numbers still hold under that treatment, the claim is probably true, and it is certainly more useful than a single impressive run that nobody can repeat.

Where this does not apply

On systems where the data always fits in memory and is always hot (a database buffer pool, a cache server), warm numbers are the real ones. For network benchmarks, cold means including DNS and connection setup, which real clients mostly pay once. Containers share the host's page cache, so a benchmark inside one is affected by what ran outside. See the kernel documentation of drop_caches and the fincore manual page.