How to Replace a Bash Backup Script With Python the Safe Way

Most backup scripts start as three lines of Bash and grow into forty, with variables that must be quoted, a find to delete old files and no way to tell a failure from success. The point where it is worth rewriting in Python is when error handling starts to dominate. This post shows a compact version, comparing it to rotating backups in shell. The examples assume Python 3.13.

The script

#!/usr/bin/env python3
import logging
import sys
import tarfile
from datetime import datetime, timezone
from pathlib import Path

SOURCE = Path("/srv/data")
DEST = Path("/var/backups/data")
KEEP = 7

log = logging.getLogger("backup")


def make_archive(source: Path, dest: Path) -> Path:
    stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
    final = dest / f"data-{stamp}.tar.gz"
    partial = final.with_suffix(".gz.partial")
    with tarfile.open(partial, "w:gz") as tar:
        tar.add(source, arcname=source.name)
    partial.rename(final)
    return final


def prune(dest: Path, keep: int) -> None:
    archives = sorted(dest.glob("data-*.tar.gz"))
    for old in archives[:-keep]:
        log.info("removing %s", old.name)
        old.unlink()


def main() -> int:
    logging.basicConfig(level=logging.INFO, format="%(asctime)s %(message)s")
    DEST.mkdir(parents=True, exist_ok=True)
    try:
        archive = make_archive(SOURCE, DEST)
        log.info("wrote %s (%d bytes)", archive.name, archive.stat().st_size)
        prune(DEST, KEEP)
    except (OSError, tarfile.TarError) as err:
        log.error("backup failed: %s", err)
        return 1
    return 0


if __name__ == "__main__":
    sys.exit(main())

2026-11-03 02:30:01,114 wrote data-20261103T023001Z.tar.gz (48211904 bytes)
2026-11-03 02:30:01,115 removing data-20261027T023001Z.tar.gz

The output shown is representative of the log format. The timestamp in the name sorts lexically, so a plain sorted() orders archives by age without parsing dates. Writing to a .partial name and renaming means no reader ever sees a half-written archive, because rename within one file system is atomic.

What Python fixes over Bash

Paths are objects, so spaces in names are not a hazard. A failing tar in a pipeline is hidden unless pipefail is set, while here TarError and OSError are caught explicitly and turned into a non-zero status. Retention is arithmetic on a list, not ls -t | tail -n +8 | xargs rm, a line that breaks on unusual names. Logging goes through one configured logger, and cron can mail or capture the exit status. The pattern is the same as in the shutil guide.

Handle the details

def free_bytes(path: Path) -> int:
    import shutil
    return shutil.disk_usage(path).free


if free_bytes(DEST) < 2 * sum(p.stat().st_size for p in SOURCE.rglob("*") if p.is_file()):
    log.error("not enough free space for a backup")
    sys.exit(1)

Check free space before writing, exclude directories with a filter function passed to tar.add, and encrypt the result before it leaves the host if the data is sensitive. For databases, call the dump tool with subprocess.run(..., check=True) first and archive its output. Schedule the script with cron or a timer; see scheduling a Python script, and prove it restores using backup verification.

Making the archive trustworthy

A backup is only useful if it can be read back later, so the script should check its own work before it reports success. Write the archive to a temporary name in the destination directory, compute a SHA-256 digest while streaming it, and rename it into place only after the write finishes. A reader scanning the directory then never sees a half-written file, and the digest stored beside the archive lets a later check prove that the bytes have not changed.

Re-reading the archive after writing it catches a class of failure that a clean exit status hides: a full disk can truncate output in ways some tools do not report, and a flaky network share can corrupt a copy. tarfile.open followed by listing every member is a cheap integrity check for a compressed tar. It does not prove the data is right, only that the container is readable, so it complements a periodic restore test, described in verifying that a backup restores.

Retention without deleting the wrong thing

Pruning is the dangerous half of a backup script. A bug in the retention code can delete every archive, including the one just made. Guard it with rules that fail safe: only consider files whose names match the exact pattern the script generates, never delete the newest archive, and refuse to delete when the number of matches is fewer than the number you intend to keep. Sorting by the timestamp in the name is safer than sorting by modification time, since a copy or restore can change mtimes.

A common policy keeps seven daily, four weekly and twelve monthly archives. The logic to select them is a few lines of date arithmetic, and it is worth putting under a unit test with a made-up list of file names, because an off-by-one there is invisible until the day you need the file that was pruned. The simpler policy covered in rotating backups may be enough for a small system.

Permissions, locks and scheduling

Backups contain everything, including secrets, so create the archive with mode 0600 from the start. Calling os.umask(0o077) at the top of the script, or opening the file with os.open and an explicit mode, avoids a window where the file exists with default permissions. Keep the destination on a different disk or host from the source; a copy on the same device protects against deletion but not against a failed disk.

Prevent overlapping runs with a lock file taken through fcntl.flock, which the kernel releases when the process dies, so a crash cannot leave a stale lock. Exit with a non-zero status on any failure and send the failure somewhere a person will see it. Schedule the script with cron or a systemd timer, as in scheduling a Python script, and add a dead-man check that alerts when no backup has succeeded in the expected period. Silence is the most common backup failure, and only an alert on the absence of a success will catch it.

Restoring is the real test

Everything above makes it likelier that an archive is good. Only a restore proves it. Build the restore step into the same program as a second command, so it receives the same maintenance as the backup code. It should extract into an empty temporary directory, compare file counts and a sample of checksums against the manifest written at backup time, and print a clear pass or fail.

Run it automatically on a schedule, not only when disaster strikes. A weekly job that restores the latest archive into a scratch directory and checks it finds the failures that matter: a path excluded by mistake, a permissions problem, a database dump that does not load. Record the result in the same place as the backup status so one alert covers both.

Write down the manual procedure too, including where the keys for encrypted archives live and who can reach them. An archive encrypted with a key stored only on the machine that failed is not a backup. Store the key separately, test that it decrypts, and review the document whenever the script changes, because a restore guide that describes last year's layout is only slightly better than none.

Where this does not apply

If the whole job is rsync -a --delete /srv/data/ host:/backups/data/, a shell one-liner in cron is the right size, and the Python version adds nothing. Python also needs an interpreter on the machine, which a minimal container may not have. For large trees, shelling out to tar with zstd is faster than tarfile in one thread. See the tarfile documentation and the pathlib documentation.