How to Debug High Load With top vmstat and iostat Safely

A load average of 12 sounds alarming and means nothing until you know how many cores the machine has and what the processes are waiting for. A load that is high with idle CPUs, which surprises people, is usually a disk or network filesystem problem. This post is a short decision tree with the commands that settle each branch. It continues the first-look tools from Linux troubleshooting and sets up profiling with perf. Examples use procps 4.0 and sysstat 12.7 on Debian 13.

Step 1: the numbers

uptime
nproc

 10:42:11 up 41 days,  3:12,  2 users,  load average: 11.80, 9.42, 6.10
4

The three values are the averages over 1, 5 and 15 minutes. On four cores, a sustained 4 means every core is busy and anything above means queueing. Here the load is rising (11.8 against 6.1 fifteen minutes ago) and about three times the core count. On Linux the figure includes processes in uninterruptible sleep, usually waiting for IO, so load can be high on an idle CPU.

Step 2: where the CPU time goes

top -b -n 1 | head -n 12

%Cpu(s):  3.1 us,  1.2 sy,  0.0 ni, 12.4 id, 82.8 wa,  0.0 hi,  0.1 si,  0.4 st

Read the CPU line. High us and sy is compute: look at the top processes by CPU. High wa, as above, means the CPUs are idle because processes wait for the disk. High st is steal, a virtual machine starved by its host. High si points at network interrupts. In interactive top, press 1 for per-core numbers and P or M to sort by CPU or memory.

Step 3: vmstat for the shape

vmstat 1 5

procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu-------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1 10  81920  41200   9210 804212  220  540  9100  7800  900 1400  3  1 12 83  0

r is the number of runnable processes and b the number blocked in uninterruptible sleep: here 10 blocked and 1 running, so the load is disk wait. si and so are pages swapped in and out per second; the non-zero values mean memory pressure, where the machine is thrashing: swapping makes everything wait for the disk, which then shows up as wa. Discard the first line of output, which is an average since boot. Related memory reading is in how much swap a server needs.

Step 4: which disk, which process

iostat -xz 1 3
ps -eo stat,pid,user,wchan:20,cmd | awk '$1 ~ /^D/' | head
sudo pidstat -d 1 3

Device   r/s    w/s  rkB/s  wkB/s  r_await  w_await  aqu-sz  %util
sda     310.0  120.0 9100.0 7800.0   42.10   95.30   18.2   99.8

%util near 100 with a growing await (average milliseconds per request) means the device is saturated. aqu-sz is the queue length. The ps filter lists processes in state D and the kernel function they sleep in (wchan), such as io_schedule or an NFS wait, and pidstat -d attributes disk reads and writes to processes. iotop -oPa offers a live view. An NFS server that stopped answering shows as a pile of D-state processes and no local disk activity.

Step 5: pressure stall information

cat /proc/pressure/cpu /proc/pressure/io /proc/pressure/memory

some avg10=4.12 avg60=2.80 avg300=1.30 total=18322311
some avg10=71.40 avg60=66.20 avg300=48.10 total=5582104411
full avg10=64.00 avg60=60.10 avg300=44.80 total=5210488832

Kernel pressure stall information (PSI, in Linux 4.20 and later) measures the share of time tasks were stalled on each resource. It answers directly which resource is the bottleneck, and is a better basis for alerts than load average. High io values, as above, confirm the diagnosis from the earlier steps.

What to do with the answer

• CPU-bound: find the process with top, profile it, or add capacity; nice and cgroups limit the damage meanwhile, as in ionice and nice.
• disk-bound: find the writer with pidstat -d, stop or throttle it with ionice, check for a failing disk with smartctl.
• Memory pressure: find the largest resident processes with ps aux --sort=-rss | head, fix the leak or add RAM.
• Steal: ask the hosting provider, or move the workload.

Where this does not apply

Containers see the host's load average, not their own, so use cgroup statistics (cat /sys/fs/cgroup/cpu.stat) for per-container diagnosis. A short spike during a deploy or a backup is normal. Record numbers over time before judging; see capacity checks and node exporter.

What the load average counts

The three numbers from uptime are averages over one, five and fifteen minutes of the number of tasks that are running, waiting to run, or in uninterruptible sleep, most often waiting for disk. A load of 4 on a machine with 4 cores means the cores are busy on average, and a load of 4 on a single core means three tasks are queuing. Compare the figure with the number of cores shown by nproc, and look at the trend across the three values: a one-minute number far above the fifteen-minute one means the load is recent and rising. Because tasks stuck on slow storage count, a high load does not necessarily mean the CPU is busy. A machine with a hung network mount or a failing disk can show a load of 50 with the processor idle. That is why the first question is not how high the number is, but what the tasks are waiting for, and the other tools answer it.

In a virtual machine, add one more question: is the host overcommitted? The steal column in top shows time your virtual CPU wanted to run and the hypervisor did not allow, and a high value points at the host and not at your processes.

Telling CPU from disk from memory

top breaks CPU time into user, system, idle, wait and steal percentages. High user time means programs computing, high system time means the kernel doing work for them, such as network handling and system calls, and high wait means CPUs idle while waiting for storage. vmstat 1 prints a line per second and shows the shape over time: r is the number of runnable tasks, b the number blocked, si and so memory swapped in and out, bi and bo blocks read and written, and the CPU columns again. A queue of runnable tasks with no idle time is a CPU problem. Blocked tasks with high wait are a storage problem. Steady swap activity is a memory problem, which produces disk waits as a side effect. Ignore the first line of vmstat output, which is the average since boot and not the current state. Watch for a few intervals before concluding, since a spike in one second proves little.

Finding the disk and the process

For disks, iostat -x 1 from the sysstat package shows per-device utilization, request rates and the time requests wait. A device near 100 percent utilization with rising wait times is saturated, though on devices that handle many requests in parallel, such as NVMe drives and arrays, the utilization figure overstates the problem, so look at the latency and queue columns instead. To see which process is responsible, pidstat -d 1 lists reads and writes per process, and iotop does the same interactively if you can run it as root. Processes stuck on disk show a state of D in ps output, and listing them with ps -eo state,pid,cmd | awk '$1 == "D"' names what is blocked. On kernels with pressure stall information, the files in /proc/pressure/ report how much time tasks spent stalled on CPU, memory and disk, as short-term averages, and they are a clearer measure of contention than the load average. A value on the io line near zero rules disks out at once.

Answers lead to actions. For a CPU-bound process, profile it or limit it with nice or a cgroup. For storage, look at what is writing and when. For memory, find the process that is growing or the limit that is too low.

See the kernel PSI documentation and vmstat(8).