Find syscalls that take time
Show how long each system call takes with strace -T. The figure in angle brackets is the time spent inside the call, so a slow write or sleep stands out.
A command seems to hang or run slowly, and you want to know which system call the time goes to.
strace -T -e trace=clock_nanosleep sleep 0.2 2>&1 | head -n 1
strace -T -e trace=write dd if=/dev/zero of=out bs=50M count=1 2>&1 | grep -E '^write\(1,' | grep -oE '= [0-9]+ <[0-9.]+>'clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=0, tv_nsec=200000000}, 0xffffc1e5a988) = 0 <0.200338>
= 52428800 <0.190752>-T appends the time spent in each traced call, in seconds, in angle brackets at the end of the line. The sleep call shows the 0.2 seconds it was asked for, plus a little overhead. The write shows how long the kernel needed to copy 50 MB, which is where the time of a copy command goes when the disk or the page cache is slow. Pair it with -e trace= to keep only the calls of interest. See the strace manual.
The figure is wall-clock time inside the call, including time spent blocked or descheduled, and it is not CPU time. The tracing overhead makes short calls look longer than they are. Use -c for totals and -r for the gap between calls. Without -f the children of a command are not traced.