Compress with Multiple Cores Using zstd -T0 or pigz
Compress on all cores with zstd -T0 or pigz. On 27 MB of Python source, zstd level 12 took 1.26 s on one thread and 0.82 s on ten, with an identical output size.
A backup spends most of its time compressing on a single core while the others sit idle.
tar cf src.tar /usr/lib/python3.11 2> /dev/null
echo "input: $(stat -c %s src.tar) bytes, $(nproc) cores visible"
TIMEFORMAT='%Rs'
echo -n "zstd -T1 -12: "; time zstd -q -f -T1 -12 src.tar -o one.zst
echo -n "zstd -T0 -12: "; time zstd -q -f -T0 -12 src.tar -o all.zst
echo -n "gzip -6: "; time gzip -6 -c src.tar > src.gz
stat -c '%n %s' one.zst all.zst src.gzinput: 27566080 bytes, 10 cores visible
zstd -T1 -12: 1.260s
zstd -T0 -12: 0.821s
gzip -6: 0.729s
one.zst 6261917
all.zst 6261917
src.gz 7947946-T0 tells zstd to start as many worker threads as there are CPU cores, and -T1 forces one. The tarball of the Python standard library, 27566080 bytes, compressed in 1.26 seconds with one thread and 0.82 with ten, and both outputs were 6261917 bytes. gzip -6 was faster at 0.73 seconds and produced 7947946 bytes. pigz applies the same idea to gzip, and is not installed in this container, so it was not timed. See the zstd manual.
The speedup was 1.5 times on ten cores, because a small input is split into few jobs and the tar step is serial; large inputs scale better. Each thread uses extra memory, and a busy machine may gain nothing. Compare with gzip on your own data before switching, since a decompressor must exist on the other side.