Speed Up sort with a Faster Temp Directory on Linux
Choose where sort writes its temporary files with sort -T dir, or the TMPDIR variable, when the input outgrows memory. strace shows the files land in that directory.
Sorting a huge file fills the small /tmp, or you want the temporary files on a faster disk.
seq 1 300000 | shuf > nums.txt
mkdir spill
strace -f -e trace=openat -o st.txt sort -S 1M nums.txt > /dev/null
echo "default: $(grep -c '"/tmp/sort' st.txt) opens under /tmp"
strace -f -e trace=openat -o st.txt sort -S 1M -T spill nums.txt > /dev/null
echo "with -T spill: $(grep -c '"spill/sort' st.txt) opens under spill/"
grep -o '"spill/sort[^"]*"' st.txt | sort -u | head -n 2default: 82 opens under /tmp
with -T spill: 82 opens under spill/
"spill/sort0A5TG5"
"spill/sort0NcMgs"When the data is larger than the memory buffer, sort writes sorted runs to temporary files and merges them. -S 1M shrinks the buffer to force that on a small input. By default the files go to /tmp, and the trace counted 82 opens of paths like /tmp/sortXXXXXX; with -T spill the same sort opened its files inside spill/. TMPDIR has the same effect, and -T can be repeated to spread files over several disks. See sort invocation.
This run shows where the files go and does not measure speed, so time your own data before assuming a gain. A larger -S avoids temporary files altogether when memory allows. The directory must have room for roughly the size of the input, and a RAM-backed tmpfs just moves the same data into memory.