How to Test Idempotency of a Provisioning Script Safely

A provisioning script gets run more than once in practice: after a failure, after an edit, by a second person, by a scheduler. If the second run breaks something, appends the same line again or fails on "already exists", the script is a one-shot and a liability. Idempotency can be tested, and the test is short. This post shows the snapshot method and the common fixes. It complements safe, idempotent deploys and sandboxing destructive scripts. Examples use Bash 5.2 and a Debian 13 container.

The test harness

#!/usr/bin/env bash
set -euo pipefail

snapshot() {
  {
    find /etc/myapp /opt/myapp -type f -exec sha256sum {} + 2>/dev/null
    find /etc/myapp /opt/myapp -printf '%m %u %g %p\n' 2>/dev/null
    dpkg-query -W -f='${Package} ${Version}\n'
    getent passwd myapp || true
    systemctl is-enabled myapp 2>/dev/null || true
  } | sort
}

./provision.sh
snapshot > /tmp/after-first.txt

out=$(./provision.sh 2>&1) || { echo "second run failed"; echo "$out"; exit 1; }
snapshot > /tmp/after-second.txt

if diff -u /tmp/after-first.txt /tmp/after-second.txt; then
  echo "idempotent: state identical after second run"
else
  echo "NOT idempotent: state changed on second run" >&2
  exit 1
fi

docker run --rm -v "$PWD":/work:ro debian:13 bash -c \
  'apt-get update -qq && apt-get install -y -qq sudo >/dev/null && cp -r /work /tmp/w && cd /tmp/w && bash tests/idempotent.sh'

The snapshot collects what the script is supposed to control: file contents by checksum, modes and owners, installed packages, a user entry and a service state. Add what your script manages. The comparison fails on any difference, and diff -u shows exactly which file changed, such as a config with a line appended twice. A fresh container makes the first run a true first run and keeps the host clean. The test cares about the second run changing nothing, which is stronger than "did not fail".

Count the changes

./provision.sh | tee /tmp/run2.log
grep -c '^CHANGED' /tmp/run2.log | grep -qx 0

If the script logs a line for each action it takes, such as CHANGED: installed nginx versus OK: nginx already installed, the second run should print no CHANGED lines. That mirrors the changed=0 line of Ansible and gives the operator the same visibility. Build the logging into the helper functions so each step reports its own result.

The failure midway

FAIL_AFTER=3 ./provision.sh || true
./provision.sh
snapshot > /tmp/after-recovery.txt
diff -u /tmp/after-first.txt /tmp/after-recovery.txt

A first run that died partway is the realistic case: a network blip, a full disk. Add a test hook, a variable that makes the script exit after N steps, then rerun and compare with a clean run. Scripts that create a directory, then a file in it, then fail, and are re-run, often break on the "already exists" error from the first step. This test finds those.

Idempotent building blocks

mkdir -p /opt/myapp/data
ln -sfn /opt/myapp/releases/1.4.0 /opt/myapp/current
install -m 644 -o root -g root app.conf /etc/myapp/app.conf
grep -qxF 'vm.swappiness=10' /etc/sysctl.d/90-app.conf 2>/dev/null || echo 'vm.swappiness=10' >> /etc/sysctl.d/90-app.conf
id myapp >/dev/null 2>&1 || useradd --system --no-create-home myapp
apt-get install -y nginx

Each of these is safe to repeat: mkdir -p succeeds if the directory exists, ln -sfn replaces a symlink in place, install overwrites with the same content, the grep -qxF ... || form appends a line only when it is missing, the user is created only when absent, and apt-get install is a no-op for an installed package. The unsafe forms: plain echo ... >> file, useradd without a check, mkdir without -p and git clone into an existing directory.

Where this does not apply

Some operations are not idempotent by nature, such as sending an email or running a database migration that is not guarded; protect those with a marker or the migration tool's own tracking. A snapshot that includes timestamps or logs will always differ, so exclude them. Real hosts have state that a container lacks, such as systemd and kernel parameters; use a VM for those. See testing install scripts for the same method applied to installers.

What idempotent means for a provisioning script

A script is idempotent when running it twice leaves the system in the same state as running it once, and the second run changes nothing. That property is what lets you rerun a script after a failure, apply it to a machine that is partly configured, and use it to correct drift without fear. It is easy to break by accident. Appending a line to a file adds it again each run. Creating a user fails the second time. Downloading and unpacking overwrites local changes. Restarting a service on every run causes needless interruptions. Idempotence is not the same as safety or correctness: a script can be idempotent and wrong, converging on the wrong state every time. What the test checks is narrower and still valuable: that the second run is a no-op, and that an interrupted run can be completed by running again. Those two properties cover the situations where provisioning scripts cause real trouble.

Declarative tools such as Ansible build the property into their modules. For shell scripts you build it yourself, step by step, and the test is what keeps you honest.

A harness that proves it

Run the script in a clean container or virtual machine, twice, and compare. After the first run, capture the state that matters: a checksum of the configuration files with find /etc -type f | sort | xargs sha256sum, the list of installed packages, the users and groups, the enabled services and the listening ports. Run the script again, capture the same, and fail if anything differs. Also check the second run's own report: have the script print one line per change it makes, with a fixed prefix, and assert that the second run printed none. That is more precise than comparing states, since it catches a step that rewrote a file with identical content, changing its timestamp and triggering a service restart, which the state comparison would miss. Include the exit status in the check: the second run must also exit zero, which catches the "user already exists" class of failure at once.

Interrupted runs

The more valuable test is the failure in the middle. Make the script stop after step three, by injecting a failure with a stub command or an environment variable that causes an exit at a chosen point, then run it normally and check that it finishes the job and ends in the same state as an uninterrupted run. Repeat for each step, which is a loop in the test and finds the steps that leave half-done work behind: a file written but not made executable, a directory created and not populated, a service enabled and not started. Write files in a way that is safe against interruption, to a temporary name followed by an atomic rename, so a partial file is never in place under its final name. Use lock files carefully, since a lock left behind by a killed run can block every later one, and have the script detect and clear a stale lock. A provisioning script that survives being killed at any point is one you can trust on a bad day.

Building blocks that are naturally idempotent

Choose commands and patterns that converge. mkdir -p succeeds if the directory exists. ln -sfn replaces a symlink in place. install -m 0644 -D sets mode and creates parents. To add a line to a file only if it is missing, use grep -qxF "$line" file || printf '%s\n' "$line" >> file, with the fixed-string and whole-line flags so the check does not mistake a pattern for text. Create users and groups after checking with id or getent. Package installs are idempotent in practice, since the package manager skips what is already present, though a version pinned in the script needs to be upgraded deliberately. Write a file only when its content would change, comparing with cmp -s, so the timestamp stays put and the dependent service is not reloaded for nothing. For services, enable and start only when the state differs, and reload only when a configuration actually changed, using a flag set when a file was updated. Each of these is a few lines, and together they turn a fragile script into a reliable one.

See Ansible's definition of idempotency and ln(1).