Skip to content
shellhacks.dev
DevOps

How to Do Capacity Checks With Shell Commands the Right Way

Run capacity checks from the shell: disk and inode use, memory available, load against core count, a growth estimate and a simple CSV log for trends.

6 min read
Summarize in AI
Illustration for "How to Do Capacity Checks With Shell Commands the Right Way": a terminal session, a disk usage gauge, a scrolling log.

An outage from a full disk is a failure of foresight, because the disk filled slowly and visibly for weeks. A short script that records the key numbers each day, and calculates when the limit will be reached, replaces surprise with a ticket. This post covers the commands and a small log. It sits between alerting and the full metrics setup in node exporter. The commands assume Debian 13 with procps-ng 4.0.

The four numbers

Shell
df -h --output=target,size,used,avail,pcent / /var
df -i --output=target,itotal,iused,ipcent /
free -m
echo "load: $(cut -d' ' -f3 /proc/loadavg), cores: $(nproc)"
Text
Mounted on  Size  Used Avail Use%
/            40G   31G  7.2G  82%
/var         20G   11G  8.1G  58%
Mounted on  Inodes IUsed IUse%
/           2621440 412881  16%
               total  used  free  shared  buff/cache  available
Mem:            3911  1210   310      14        2390       2503
load: 0.42, cores: 4

Disk blocks and inodes are separate limits; a filesystem full of tiny files runs out of inodes at 20 percent block use, as in debugging disk full and inode exhaustion. Memory headroom is the available column, which includes reclaimable cache; free understates it. Load is meaningful only against the core count: a load of 4 on four cores is fully busy, on 32 cores it is idle.

Log it daily

/usr/local/bin/capacity-log.shShell
#!/usr/bin/env bash
set -euo pipefail
log=/var/log/capacity.csv
[ -s "$log" ] || echo "date,root_used_kb,root_size_kb,mem_avail_mb,load15,cores" > "$log"

read -r size used < <(df --output=size,used -k / | tail -n1)
avail=$(free -m | awk '/^Mem:/ {print $7}')
load15=$(cut -d' ' -f3 /proc/loadavg)

echo "$(date +%F),$used,$size,$avail,$load15,$(nproc)" >> "$log"

A CSV is the simplest store: readable with a spreadsheet, awk or column -t -s,. Run it from cron once a day. For heavier needs a time-series database replaces it, but the script costs nothing and survives anything.

Estimate days until full

Shell
awk -F, 'NR>1 {d[NR]=$1; u[NR]=$2; s=$3}
  END {
    n=NR; days=n-2
    if (days<1) exit
    rate=(u[n]-u[2])/days
    if (rate<=0) { print "not growing"; exit }
    print int((s-u[n])/rate) " days until full at " int(rate/1024) " MB/day"
  }' /var/log/capacity.csv
Text
38 days until full at 210 MB/day

The estimate assumes one row per day and linear growth, which is rough but catches the case that matters: a steady climb. Add a threshold, for example warn when fewer than 21 days remain, and send it with the alert channel you already have. A spike followed by a cleanup produces a misleading rate, so compare over a longer window, such as the last 30 rows, when you have them.

Where this does not apply

Linear projection fails for workloads with cycles or bursts, such as month-end batches. Cloud volumes that resize automatically need a limit check on cost instead. Memory and CPU need percentile views over time to size correctly, not one sample a day; metrics systems do this. Databases carry their own capacity limits (connections, table size); see checking database size. For the quick tools, review df and du.

Reading each number correctly

Each of the four numbers has a trap. For disk, df reports space by file system, and a full disk can hide in two places that the usual command misses. Inodes run out when a directory holds millions of tiny files, and the inode report of df shows it with plenty of bytes free. Deleted files that a running process still holds open keep their space until the process closes them, so df and du disagree, and lsof +L1 lists those files. For memory, the figure that matters is available memory, which free prints in a column of its own, since the kernel uses free memory for cache and returns it when programs ask. A machine with little free and a lot available is healthy. For CPU, the load average counts processes that are running or waiting, including those blocked on disk, so compare it with the number of cores and look at the disk wait share to tell computing from waiting.

Swap activity is the fourth signal. Used swap is not a problem by itself. Pages moving in and out constantly, shown in the si and so columns of vmstat, is.

A single reading says little, and a series says a lot. Have a small script run daily from cron or a timer and append a line with the date and the numbers: disk percent for each important mount, available memory, load, and the count of failed units. Use a stable text format such as tab-separated columns, so the file is easy to read, graph or load into a spreadsheet. Keep the history for a year, which is tiny. After a few weeks, the numbers show the growth rate, and you can answer the question the checks exist for: how long until this fills. Compare the same day of the week, since many systems have weekly cycles, and note anything that changes the pattern, such as a deployment or a new customer, in a comment line so you can explain a step in the data later.

Alert on the trend as well as the level. A script can compute the growth over the last seven days and print the estimated date of exhaustion, and warn when it is under a month away.

Turning numbers into time left

The most useful figure is days until full. Take the used space now and a week ago, divide the difference by seven for the daily growth, and divide the free space by that. The estimate assumes steady growth, and real growth is lumpy: a monthly batch job or a log burst can fill a disk much sooner than the average suggests. Use the largest weekly growth you have seen for a cautious figure. Add a safety margin, since file systems perform badly when nearly full, and some reserve a percentage for root, so the effective free space for ordinary users is smaller than the raw number. When the estimate drops below a threshold such as thirty days, open a ticket, and below seven, send a page. Then act on it by cleaning, compressing, moving data or adding space, and record which, so the history explains the sudden drop in usage later.

A script you can run today

A first version needs only a few lines. Print the date, the used percentage of each real mount from df -P, available memory in megabytes from free -m, the one-minute load from /proc/loadavg, and the count of failed systemd units from systemctl --failed. Append the line to a file under /var/log from a daily timer. Use df -P for its fixed output format, since the default wraps long device names onto two lines and breaks parsers. Exit with a nonzero status when any value crosses a limit, and let the scheduler report it. From there add a mail or push message for the cases you care about. Keep the thresholds in variables at the top of the script and the reasons in comments, so the numbers can be adjusted with understanding and not by guesswork.

Be careful with network file systems. A hung NFS mount can make df block forever, which turns the capacity check into a hung job. Run it with a timeout, and exclude remote file systems with the type filter if they are monitored elsewhere.

See the kernel's meminfo documentation for MemAvailable and df(1).

Share

Keep reading

Get new posts by email

One email when a new post is published. No spam.

Unsubscribe in one click. No spam, ever.