How to Run Jobs Only When the System Is Idle in Real Scripts

Backups, indexing and cleanups should not compete with the work the machine is there for. "Run it when idle" has two meanings, and the tools address them differently: run it at low priority so it yields, or do not start it unless the system is quiet. Both are cheap to add to a cron job. This builds on scheduling with cron, and the commands assume util-linux 2.40 and Linux 6.x.

Lower the priority

nice -n 19 ionice -c3 tar -czf /var/backups/home.tar.gz /home
ionice -p $$

idle

nice -n 19 gives the CPU scheduler the lowest weight, so other processes get the CPU whenever they want it. ionice -c3 puts disk reads and writes in the idle class: the job gets disk time only when no other process asked for it in the recent past. The idle IO class works only with schedulers that honour it (BFQ, and mq-deadline partially), so on a system using none for NVMe the effect is limited. Check the scheduler in /sys/block/sda/queue/scheduler.

Skip the run when busy

#!/usr/bin/env bash
set -euo pipefail
max_load=1.5
read -r load _ < /proc/loadavg
if awk -v l="$load" -v m="$max_load" 'BEGIN { exit !(l > m) }'; then
  echo "load $load above $max_load, skipping" >&2
  exit 0
fi
exec nice -n 19 ionice -c3 /usr/local/bin/reindex.sh

The first number in /proc/loadavg is the one-minute load average. Compare it with a threshold chosen for the machine, such as half the number of cores. A job that skips silently might never run on a permanently busy server, so log the skips and count them: if it skipped more than a few times in a row, alert. The comparison uses awk because Bash cannot compare decimals. See scripts that behave differently on reruns for why the skip path needs to be quiet but visible.

In a systemd unit

[Unit]
Description=Nightly reindex

[Service]
Type=oneshot
ExecStart=/usr/local/bin/reindex.sh
Nice=19
CPUSchedulingPolicy=idle
IOSchedulingClass=idle
CPUQuota=50%

[Timer]
OnCalendar=*-*-* 02:30:00
RandomizedDelaySec=30min
Persistent=true

[Install]
WantedBy=timers.target

CPUSchedulingPolicy=idle uses the kernel's SCHED_IDLE policy, which is even lower than nice 19. CPUQuota caps the total CPU time, useful when idle priority is not enough. Pair the unit with a timer, as in systemd timers with OnCalendar. The directives are described in systemd.exec.

Idle on a desktop

xprintidle
loginctl show-session "$XDG_SESSION_ID" -p IdleHint -p IdleSinceHint

On a desktop, "idle" often means the user has not touched the keyboard. xprintidle reports milliseconds of X11 idle time, and logind exposes an idle hint. Cron runs outside the session and cannot see these without the right environment variables, so desktop jobs fit better in a user systemd unit or a login-time script. Waiting for idleness before a laptop starts a backup is why anacron exists, see anacron.

What priority does and does not control

A niceness value changes how the scheduler divides CPU time between runnable processes, and nothing else. A process at niceness 19 still holds its memory, its open files and its locks, and it can still saturate the disk if no one else is using it. That is the intended behaviour for a backup that should use spare capacity, and it is the source of the surprises: the job is polite about the CPU while remaining a heavy consumer of memory and IO.

The IO side is subtler. The idle class in ionice only guarantees that the job waits when another process is actively reading or writing, and it depends on the block scheduler. On a fast NVMe drive running with no scheduler, the request queue is so short that priority has little to act on. If a nightly job still hurts interactive latency, the fix is a cap rather than a priority: a cgroup limit on read and write bandwidth, or systemd-run --property=IOReadBandwidthMax= on a particular device, bounds the damage regardless of what the scheduler does.

Testing that a job really stays out of the way

Claims about low impact deserve a measurement. Start the job on a test machine, run a latency probe against the main workload, for example a loop of small requests to the local web server that records response times, and compare the 95th percentile with and without the job running. If the percentile rises noticeably even at the lowest priority, the contention is on something priority does not govern, such as memory bandwidth, the page cache or a shared lock.

Page cache pressure is the usual hidden cost. A backup that reads every file on a server pushes the web application's hot files out of memory, and the first requests after the job finish are slow because they must reload from disk. Tools that read with the O_DIRECT flag or that call posix_fadvise with DONTNEED avoid this, and some archive programs have an option for it. Without such an option, the cost is acceptable only if the job runs when traffic is lowest.

Choosing the time as well as the priority

Idle detection is a fallback for when the schedule cannot be chosen. If the traffic pattern is known, a fixed quiet window is more predictable than an idle check, because the job finishes at a known time and a person can reason about what it overlapped with. A timer with a randomised delay, as in the unit shown earlier, spreads the start across a window so that a fleet of machines does not hit a shared storage server at the same second.

Finally, give the job a way to be stopped. A long task that ignores termination signals ends up killed by the kernel or by an impatient administrator in the middle of a write. Trap the signal, finish the current unit of work, remove temporary files and exit with a status the scheduler can report. A job that can be stopped cleanly is one that can safely be given low priority, because it is expected to be interrupted often.

Checking whether the job actually ran

A job that exits early when the machine is busy needs a record of what it decided. Write one line per run to the system log with logger -t reindex "skipped, load 2.31" or "ran, 412 s", so the journal can answer how often the job executed over the last month. Without that record, a quiet failure looks the same as a quiet success until the day someone notices the index is three weeks old.

Pair the skip rule with a limit on how long a job may be postponed. A common design is a marker file updated on every successful run; when the file is older than a threshold, such as two days, the job ignores the load check and runs anyway, or raises an alert. This keeps the courtesy to other workloads from turning into starvation of the job itself, and it makes the policy explicit: run when idle, but never later than this.

Finally, test the skip path on purpose. Start a CPU-bound loop on every core, run the script and confirm that it exits with status 0 and logs the skip, then stop the loop and confirm that the next run proceeds. The branch that runs rarely is the one that hides mistakes, such as a comparison written the wrong way round that skips on a quiet machine and runs on a busy one.

Where this does not apply

Priority is not a guarantee. A low-priority job holding a database lock can block high-priority queries, and one that fills the page cache can evict hot data, as described in the page cache. Memory-heavy jobs need a cgroup limit, MemoryMax=, not a nice value. Containers see the host's load but may be limited by their own CPU quotas. See the ionice manual page and the nice manual page.