How to Write Idempotent Provisioning Scripts Step by Step

A provisioning script that works once and breaks on the second run is a fresh-machine tool only, and every later fix means a rebuild. Idempotence turns it into one you can rerun after any change. The pattern is test, then change, and it applies to a shell script as much as to a configuration tool. The examples use Bash 5.2 on Debian 12; the concept is described in what idempotency means for deploys and idempotent shell scripts.

Packages, directories, links and users

#!/usr/bin/env bash
set -euo pipefail
changed=0
note() { echo "changed: $*"; changed=1; }

pkg() {
  dpkg -s "$1" >/dev/null 2>&1 || { apt-get install -y -qq "$1" >/dev/null; note "installed $1"; }
}
pkg nginx
pkg git

[[ -d /srv/app ]] || { mkdir -p /srv/app; note "created /srv/app"; }

id appsvc >/dev/null 2>&1 || { useradd --system --shell /usr/sbin/nologin appsvc; note "user appsvc"; }

[[ $(readlink /srv/app/current 2>/dev/null) == /srv/app/releases/1.4 ]] \
  || { ln -sfn /srv/app/releases/1.4 /srv/app/current; note "current -> 1.4"; }

Each block asks "is it already right?" before acting, and records a change only when it acts. apt-get install and mkdir -p are already repeatable, but the explicit check lets the script tell you what it did, and avoids needless work. ln -sfn replaces a symlink atomically enough for deploys, and -n prevents it from descending into a directory link.

Files: write only on change

put_file() {                    # put_file DEST MODE  (content on stdin)
  local dest=$1 mode=$2 tmp
  tmp=$(mktemp)
  cat > "$tmp"
  if [[ -f $dest ]] && cmp -s "$tmp" "$dest" && [[ $(stat -c %a "$dest") == "$mode" ]]; then
    rm -f "$tmp"
    return 1                    # unchanged
  fi
  install -m "$mode" "$tmp" "$dest"
  rm -f "$tmp"
  note "wrote $dest"
}

put_file /etc/nginx/conf.d/app.conf 644 <<'CONF' && systemctl reload nginx
server { listen 80; location / { proxy_pass http://127.0.0.1:8080; } }
CONF

The function compares the new content with the existing file and replaces it only when the content or mode differs, and its non-zero return on "unchanged" lets && systemctl reload nginx run only after a real change. Reloading on every run would be harmless for nginx and disruptive for services where a restart drops connections. Use install rather than cp so the mode is set in one step, and quote the here-document delimiter so no shell expansion changes the content.

Services and the final report

systemctl is-enabled --quiet nginx || { systemctl enable nginx; note "enabled nginx"; }
systemctl is-active --quiet nginx  || { systemctl start nginx;  note "started nginx"; }
(( changed )) || echo "no changes"

The final line is the test: run the script twice and the second run prints only no changes. If it does not, some check is wrong. Add a --check flag that makes note print and skip the action, so the script can show its plan, in the spirit of dry-run flags.

Things that cannot be made idempotent by checking

Appending a line to a file needs grep -qxF line file || echo line >> file. Database creation needs CREATE DATABASE IF NOT EXISTS or a catalogue query. Anything that sends a message, charges a card or consumes a one-time token needs a record that it was done. Use flock so two concurrent runs do not interleave, as covered in locking with flock.

What idempotent really means for each resource

An operation is idempotent when running it any number of times leaves the system in the same state as running it once. For provisioning, the stronger and more useful property is convergence: whatever state the machine starts in, running the script moves it to the desired state. That is a larger claim, because a machine that was half configured, hand edited or partly broken must also end up right. Writing for convergence means each step describes the end state and detects the current one, instead of assuming a clean start. A step that only runs useradd fails when the user exists and does nothing about a user that exists with the wrong shell. A convergent step checks the shell and corrects it.

Test convergence deliberately. Run the script on a fresh container, then again and expect no changes. Then break the machine in controlled ways, delete a file the script manages, change a mode, stop a service, edit a configuration line, and run the script a third time to check that it repairs each one. The second kind of test finds steps whose check is wrong, such as one that tests for the existence of a file and never its content.

Detecting change cheaply and correctly

The check must be accurate and cheap. Comparing file contents with cmp, modes with stat and ownership with stat -c %U:%G is exact. Package state with dpkg -s or rpm -q is quick, but checks only that a package is installed and not that it is the wanted version, so add a version comparison when it matters. For configuration lines in shared files, such as sshd_config, manage a drop-in file in the directory that the service includes, and leave the vendor file alone. That avoids editing with sed, which is the most fragile kind of idempotent step. Where an edit is unavoidable, anchor the pattern to the whole line, replace it if present, and append if absent, and show the resulting line in the report.

Ordering, dependencies and failure

Steps depend on each other: a service cannot start before its package is installed and its configuration written. Keep the order explicit in the script and avoid hidden dependencies on files created by earlier runs. When a step fails, stop and report, since later steps built on it make the situation worse. Under set -e, make sure the helper functions return the right statuses, as the status of "unchanged" in the example must not be treated as an error. Restart services only when their configuration changed, using the flag from the helper, so that a second run does not restart a production service for no reason.

Dry runs and drift detection

A --check mode that prints what would change and exits with a non-zero status when something differs turns the script into a drift detector. Run it from a scheduler and alert on a non-zero result: that finds a hand edit within a day. To implement it, have each helper take a global flag and, instead of acting, only report. Keep the report format stable so a monitoring system can parse it. This is the point at which many teams find the script does not cover enough, and the usual response is to adopt a configuration management tool, which has these features built in. Until then, a script with checks, a report, and a drift run is a respectable stopgap.

Handling secrets and generated values

Provisioning often creates values that must stay stable, such as a database password or a TLS key. Generate them only if they do not exist, store them with tight permissions, and never regenerate them on later runs, or every run rotates credentials and breaks the services that use them. A step like [[ -s $key ]] || openssl rand -hex 32 > "$key" with a prior umask 077 is the shape. Mark such files in the report as created, and as present on later runs, without printing the content. If the value must reach another host, hand it over through a secrets manager and not through the script's output.

Where this does not apply

Past a few dozen resources, or across many machines, a shell script becomes a poor imitation of a configuration management tool; Ansible, Chef or Puppet already track state, ordering and dependencies, see Ansible versus shell scripts. Scripts that must also remove things no longer wanted need a definition of "wanted", which is the point where desired-state tools pay off. See the Ansible glossary entry on idempotency and the coreutils install manual.