Count lines in a huge file fast

A file with millions of lines needs a line count, and the usual tools differ a good deal in speed.

seq 1 5000000 > big.txt
timeit() { local s=$EPOCHREALTIME e n l; n=$("$@" < big.txt); e=$EPOCHREALTIME; printf -v l '%s ' "$@"; printf '%-22s %d lines  %.2f s\n' "$l" "$n" "$(awk -v a="$s" -v b="$e" 'BEGIN{print b-a}')"; }
timeit wc -l
timeit grep -c ''
timeit awk 'END{print NR}'
timeit sed -n '$='
echo ---
printf 'a\nb' | wc -l
printf 'a\nb' | grep -c ''

wc -l                  5000000 lines  0.02 s
grep -c                5000000 lines  0.08 s
awk END{print NR}      5000000 lines  0.20 s
sed -n $=              5000000 lines  0.24 s
---
1
2

wc -l counts newline characters in large blocks and does no parsing, so it is the fastest. grep -c '' splits the input into lines, awk and sed do that and run an interpreter loop, which accounts for the order in the timings above on a 38 MB file of five million lines. The -l option is in the coreutils manual. The last pair shows the difference in meaning: wc -l counts newlines, and a final line without one is not counted.

The timings were taken on an arm64 container with Bash 5.2, coreutils 9.1 and GNU grep 3.8, and one run each, so read them as an order of magnitude. The file was already in the page cache, which a first read from disk is not. For an exact count of the lines of text including an unterminated last one, use grep -c '' or awk.