How to Write Idempotent Shell Scripts That Are Safe to Rerun

Scripts get run twice. A deploy fails halfway, someone presses the up arrow, or a cron job overlaps itself, and the script that added a line to a config file now adds it again. Writing each step as a check followed by an action prevents the damage. It is the shell version of the idea behind Ansible playbooks, and the examples assume Bash 5.2 on Linux.

Prefer commands that are already idempotent

mkdir -p /srv/app/releases
ln -sfn /srv/app/releases/20261103 /srv/app/current
install -D -m 0644 app.conf /etc/app/app.conf
rm -f /tmp/app.lock

mkdir -p succeeds if the directory exists. ln -sfn replaces an existing symlink instead of nesting a new link inside the directory it points at, which is the usual bug without -n. install -D creates parent directories and overwrites the file with the mode you set. rm -f does not complain about a missing file. Check the manual of any command you use for an equivalent.

Guard the rest with a check

ensure_line() {
  local line=$1 file=$2
  if grep -qxF -- "$line" "$file" 2>/dev/null; then
    echo "ok: $file already has the line"
  else
    printf '%s\n' "$line" >> "$file"
    echo "changed: added line to $file"
  fi
}

ensure_line "net.core.somaxconn=1024" /etc/sysctl.d/90-app.conf

changed: added line to /etc/sysctl.d/90-app.conf
ok: /etc/sysctl.d/90-app.conf already has the line

-F treats the line as a fixed string, -x demands a whole-line match and -q keeps it quiet. Without -x, a line that is a substring of another one counts as present. Printing ok or changed turns the second run into proof that the script converged.

Create things only when missing

id deploy >/dev/null 2>&1 || useradd --system --shell /usr/sbin/nologin deploy

if ! systemctl is-enabled --quiet app.service; then
  systemctl enable app.service
fi

Test the state, then act only on a difference. For users, services, packages and database rows, there is nearly always a read-only query that answers "does it exist?". For a database, use CREATE TABLE IF NOT EXISTS and INSERT ... ON CONFLICT DO NOTHING rather than a shell-side check, because the database can make the test and the change one atomic step.

Write files atomically

tmp=$(mktemp "${target}.XXXXXX")
render_config > "$tmp"
chmod 0644 "$tmp"
mv -f "$tmp" "$target"

A rename within one file system is atomic, so readers see either the old file or the new one, never half of it. Create the temporary file in the same directory as the target, since a move across file systems is a copy and loses that guarantee. Also lock the script with flock to stop two runs overlapping; preventing overlapping jobs shows how. Adding a dry run mode lets an operator see the planned changes first.

What idempotent means for a script

A script is idempotent when running it twice leaves the system in the same state as running it once. The property matters because scripts are rerun: after a failure halfway, after a timeout, by a scheduler that fires twice, or by a person who is not sure the first run finished. A script without it turns every rerun into a risk of duplicate users, doubled configuration lines, a second cron entry or an error that halts the run before the real work. A script with it can be run as often as you like, which makes recovery simple: fix the cause and run it again. Many configuration management tools exist to give this property to operations, and a shell script has to supply it by hand.

The test is easy to state. Run the script twice on a clean machine and compare the state after each run. Any difference is a bug, as is any error on the second run.

Checking state, then acting

The general pattern is to test the desired state before changing anything. If the user exists, skip creating it; if the line is in the file, do not append it; if the package is installed, move on. Use the check the platform offers: id -u name for users, grep -qxF line file for a line, dpkg -s or rpm -q for packages and test -e for files. For files whose content you manage fully, generate the whole file and replace it only when it differs, comparing with cmp -s, so the modification time stays put when nothing changed. That matters for tools that watch the file, such as services that reload on change.

Take care that the check and the action refer to the same thing. A check for a file by name, followed by an action that creates it with different contents, passes on the second run for the wrong reason.

Operations that are naturally safe

Some commands already behave well on repetition. mkdir -p succeeds if the directory exists, ln -sfn replaces a symlink atomically enough for most uses, rm -f does not complain about a missing file and install -D creates parents and sets mode and ownership in one step. Prefer these to a check followed by a plain command. Setting a value, not adding to it, is the same idea in text: write key=value by replacing the line with sed if present and appending if not, never append blindly. For systemd, systemctl enable and start are safe to repeat. Be careful with commands that look safe and are not, such as useradd, which fails when the user exists, and git clone, which fails when the directory is there.

When a command is not safe, wrap it in a function that checks first and logs what it did, using words like "created" and "already present", which make the second run's output instructive.

Partial failure and cleanup

A run that dies in the middle leaves a half-built state, and the next run must cope with it. Order the steps so that each one leaves a consistent state, and make the last step the one that makes the result visible, such as moving a finished directory into place or switching a symlink. Build in a temporary location and rename at the end. Avoid leaving lock files that survive a crash; use flock, which the kernel releases when the process exits. Record completion markers only after the step succeeds. And print a summary at the end of changes made and steps skipped, so that a clean second run shows only skips, which is the evidence that the script is idempotent. Add the double-run test to CI, in a container, and the property keeps holding as the script changes.

Where this does not apply

Some actions are inherently one-shot: sending an email, charging a card or incrementing a counter. Make those idempotent with a marker file or an idempotency key stored after success, and accept that a crash between the action and the marker repeats it. When the end state involves many resources, a tool built for it, such as Ansible or Terraform, tracks state better than a pile of guards. The coreutils manual for ln and the flock manual page document the commands used.