How to Verify Backups and Restore Procedures Automatically

Everyone has backups. Fewer have restored from them, and the first restore is often the moment of discovering that the files are empty, the archive is truncated, or the encryption key is in the lost laptop. Automating the restore test turns "we think we have backups" into evidence. It fits with verifying a backup restores, and the examples assume restic 0.17, PostgreSQL 17 and Bash 5.2.

Level 1: the backup is intact

sha256sum -c /var/backups/data-20261103.tar.gz.sha256
tar -tzf /var/backups/data-20261103.tar.gz > /dev/null && echo "archive readable"
restic check --read-data-subset=5%
borg check --verify-data repo

/var/backups/data-20261103.tar.gz: OK
archive readable
using temporary cache in /tmp/restic-check-cache-3341
check snapshots, trees and blobs
no errors were found

Checksums and archive listings detect corruption and truncation, which is cheap and catches storage problems. Backup tools verify a random subset of data per run so that the whole repository is read over a cycle. This level does not prove that the right things are in the backup or that you can restore them.

Level 2: restore into a scratch location

#!/usr/bin/env bash
set -euo pipefail
scratch=$(mktemp -d /var/tmp/restore-test.XXXXXX)
trap 'rm -rf "$scratch"' EXIT

restic restore latest --target "$scratch" --include /srv/data
live_count=$(find /srv/data -type f | wc -l)
rest_count=$(find "$scratch/srv/data" -type f | wc -l)
echo "files: live=$live_count restored=$rest_count"

# canary: a file the job writes every day
test -s "$scratch/srv/data/.canary" || { echo "canary missing" >&2; exit 1; }
# spot-check a sample against the source by hash
find /srv/data -type f -print0 | shuf -z -n 20 | while IFS= read -r -d '' f; do
  cmp -s "$f" "$scratch$f" || echo "MISMATCH $f" >&2
done

Restoring into a temporary directory proves the repository, the keys and the procedure work. A canary file written by the backup job (a timestamp or random token) proves the backup is current. The sample comparison against the live source catches silent content corruption; a tolerance is needed for files that change between backup and test, so compare against the snapshot's own recorded hashes if the source is busy. The temporary directory is removed at exit by the trap, see safe temp files.

Databases: restore and query

createdb restore_test
pg_restore --no-owner --dbname restore_test /var/backups/app-latest.dump
psql -d restore_test -Atc "SELECT count(*) FROM orders" > /tmp/restored_count
psql -d app -Atc "SELECT count(*) FROM orders WHERE created_at < now() - interval '1 day'" > /tmp/live_count
psql -d restore_test -Atc "SELECT max(created_at) FROM orders"
dropdb restore_test

The restored row count should be close to the live count of rows older than the backup, and the newest row should be recent. Pick queries that would fail if a table were empty or truncated. Run restores on a separate server or in a throwaway container, not on the production database; see throwaway databases in Docker and restore drills for small databases.

Alerting and schedule

0 3 * * *   /usr/local/bin/backup.sh         && /usr/local/bin/ping hc backup
30 4 * * 0  /usr/local/bin/restore-test.sh  && /usr/local/bin/ping hc restore-test

Run integrity checks daily and a full restore test weekly or monthly, depending on how large the data is. The test script pings a heartbeat monitor only on success, so a failed run and a missing run both alert, as described in dead man's switches. Failures should say what failed (missing canary, count mismatch) rather than just "restore failed".

Test the real recovery path

Keep the encryption key and the restore instructions somewhere that survives the loss of the server; test restoring on a fresh machine at least once a year, using only the documentation. Measure the restore time, the number that matters when the server is down. A backup that restores in nine hours when the business can tolerate two is a failed backup. Offsite copies need the same test, from the offsite location, see offsite database backups.

What a verified backup means

A backup is verified when you have shown that a restore produces the data you need, in a time you can accept, by a procedure that someone other than its author can follow. Anything less is a belief. The levels in this post build up to that: the file exists, the file is intact, the content restores, the content is right and the whole process works from a bare machine. Each level catches failures the earlier ones cannot, and the cost rises with each, so run the cheap checks often and the expensive ones on a schedule.

It helps to name the failures you are defending against. A job that silently stops running. A job that runs and backs up the wrong path after a directory was renamed. A repository that is corrupted by a failing disk. A key or password that nobody can find. A restore procedure that depends on a tool or credential that exists only on the machine that was lost. Each needs its own test, and a single "backup succeeded" message covers none of them.

Choosing what to compare

Comparing everything is slow, and comparing nothing proves little. Pick checks that would fail for the realistic failure modes. For files, compare counts and total size within a tolerance, check that the newest file's timestamp is recent, and compare checksums for a random sample of files that have not changed since the backup. For databases, compare row counts of key tables, the maximum value of a timestamp or sequence and the result of one business query, such as total orders for the previous day. For configuration, check that the restored files parse, for example by running the service's own syntax check against them.

Put a canary into the data. A small record or file containing the current time, written just before the backup starts, lets the restore test prove that the backup is fresh and from the right source. If the canary is old, the job has been backing up a frozen copy or nothing at all.

Making the test safe and cheap

Restore tests should never touch production. Use a scratch directory on a separate volume, a throwaway database container or a temporary virtual machine, and delete it afterwards. Watch disk space: restoring a large backup into the same file system that holds the live data can fill it and cause the outage you were trying to prevent. Limit the CPU and IO of the restore job with the same priority settings used for the backup itself, and run it away from peak hours. For very large datasets, restore a representative subset, such as one database or one directory tree, and rotate the subset each time so that over a quarter every part has been tested.

The human part

Technology covers the repeatable checks, and people cover the rest. Write the restore procedure as a numbered list, store it with the backup credentials in a place that survives the loss of the primary site and ask someone who did not write it to perform it once a year with a timer running. Their questions and mistakes show what the document leaves out. Record the measured time and compare it with the recovery target. If they differ, the target or the system has to change, and the time to find that out is a drill and not an outage.

Where this does not apply

A restore test cannot prove that the data was correct when backed up, or that the application will work with it. Application-level checks (running the app against the restored copy) go further for critical systems. Very large datasets may only allow sampled restores, which is better than none. See the restic documentation on checking integrity and the borg check reference.