How to Debug Disk Full and Inode Exhaustion the Safe Way

"No space left on device" comes with free space showing in some views and none in others, which makes it feel like a puzzle. There are exactly two limits (blocks and inodes), one place space hides (open deleted files), and a handful of usual suspects. This post is the order to check them in, building on the commands in df and du and the error itself. Examples use Debian 13 with ext4.

Step 1: which limit

df -h
df -i

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        40G   31G  7.2G  82% /
Filesystem      Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1      2621440 2621440       0  100% /

Here the blocks have room and the inodes are gone. Every file and directory uses one inode, so a directory with millions of small files, such as session files, mail spool entries or cache fragments, fills the inode table first. If Use% is the problem, read on; if IUse%, jump to step 4.

Step 2: find the large directories

sudo du -xh --max-depth=1 / 2>/dev/null | sort -h | tail -n 8
sudo du -xh --max-depth=1 /var | sort -h | tail -n 5

12G     /var/lib
18G     /var

-x stays on one filesystem, which prevents counting other mounts. Descend into the largest directory and repeat. ncdu / does the same interactively. Typical findings: /var/log, /var/lib/docker, /var/lib/mysql, /home, /var/cache/apt and old kernels in /boot. Clean with the owner's tool: journalctl --vacuum-size=500M, apt clean, docker system prune after checking, as in freeing up disk space.

Step 3: space held by deleted files

sudo lsof +L1 | head

COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NLINK    NODE NAME
nginx    1204 root    5w   REG    8,1 9132041216    0 1310722 /var/log/nginx/access.log (deleted)

When df says the disk is full and du accounts for much less, the difference is usually here: a log was deleted with rm while the server kept writing to it, so the file has no name but still occupies 9 GB. The space returns when the process closes the file. Reload or restart the service, or empty the file through its descriptor: : > /proc/1204/fd/5. To prevent it, truncate logs in place (: > file) or use logrotate with copytruncate or a reload hook.

Step 4: inode exhaustion

for d in /var/*; do printf '%8d %s\n' "$(sudo find "$d" -xdev 2>/dev/null | wc -l)" "$d"; done | sort -n | tail -n 3
sudo find /var/lib/php/sessions -type f -mtime +2 -delete

Count files per directory to find where the inodes went, then descend. Frequent sources are PHP session directories, mail queues (/var/spool), cache trees and an application that writes one file per event. The only cure is deleting files; free blocks do not help. Deleting millions of entries with rm in one directory is slow, and find -delete is faster. Then fix the producer so it does not recur: a cleanup job, or a different storage design.

Step 5: reserved blocks and special cases

sudo tune2fs -l /dev/sda1 | grep -i 'reserved block'
sudo tune2fs -m 1 /dev/sdb1

Ext4 reserves 5 percent of blocks for root by default, so a normal user sees a full disk while root can still write. On a large data volume, lowering the reserve to 1 percent returns space; do not do it on the root filesystem, where the reserve helps the system recover. Other places space hides: LVM snapshots filling their allocation, a bind mount hiding files underneath it (check with sudo mount --bind / /mnt/rootview; du -sh /mnt/rootview/*), and thin-provisioned storage that reports free space it does not have.

Where this does not apply

XFS and Btrfs do not have a fixed inode table; Btrfs can report "no space" from metadata exhaustion with free data space (btrfs filesystem df). Network filesystems report limits set by the server. After cleaning, add monitoring so the next fill sends a message, as in capacity checks.

Two different limits and how to tell them apart

"No space left on device" can mean two different things, and df distinguishes them. df -h reports blocks: the share of the disk's capacity that is used. the inode report of df reports inodes: the share of file entries that are used. Each file or directory needs one inode, regardless of size, and a file system is created with a fixed number of them. A directory holding millions of tiny files, such as mail queue entries, session files or cache objects, can use every inode while the disk is nearly empty by block count. The two failures look the same to a program: creating a file fails. The fix is entirely different, deleting many small files in one case and large files in the other, so check both numbers before doing anything. A third possibility is a quota on a user or a project, shown by quota or repquota, which produces the same error for one account while the disk has room.

Check the right mount, too. A directory may sit on a different file system than you assume, and df /path names the one that holds it.

Finding where the space went

To find large directories, du -xh --max-depth=1 /var | sort -h lists sizes one level down, and the -x option keeps it on one file system so a mounted volume does not distort the answer. Repeat on the largest entry, drilling down until the culprit appears. Common culprits are logs that were never rotated, the journal, package caches, old kernels, container storage under /var/lib/docker, database dumps and temporary files from a crashed job. If du accounts for much less than df reports, space is held by something it cannot see. One cause is a file deleted while a process still has it open: the name is gone, and the blocks stay allocated until the process closes it or exits. lsof +L1 lists open files with no links, and the process holding a large deleted log is usually obvious. Restarting that process frees the space, and truncating the file through /proc/PID/fd/N does it without a restart.

Another cause is data hidden under a mount point: files written to a directory before a volume was mounted over it are still on the parent file system, invisible while the mount is in place. Unmount and look, or bind-mount the parent to inspect it.

Inode exhaustion and the special cases

To find where the inodes went, count files per directory: find /var -xdev -type f | cut -d/ -f1-4 | sort | uniq -c | sort -n | tail shows the directories with the most entries. Deleting millions of files with rm -rf can take a long time, and a find ... -delete avoids the argument list limit. Prevent the recurrence with a cleanup job for the offending directory or a fix in the application that creates them. Beyond those two, a few special cases explain odd results. Ext4 reserves five percent of blocks for root by default, so a data disk fills for ordinary users while root can still write, and tune2fs -m adjusts the reserve on a volume that holds no system files. Btrfs can run out of metadata space while data space remains, and its own tools, btrfs filesystem df and usage, show it. Snapshots on LVM or Btrfs hold changed blocks, and an old snapshot may be quietly consuming a large share. The journal's size is capped with journalctl --vacuum-size, and its retention can be set permanently in the journal configuration.

See lsof(8) and tune2fs(8).