How to Make Long Running Scripts Resumable With Real Output

A migration that takes three hours and fails at the two-hour mark should not start from zero. Resumability is mostly bookkeeping: remember what finished, skip it next time, and make sure a half-finished step can safely run again. The code assumes Bash 5.2 and pairs with idempotent shell scripts.

A marker file per step

#!/usr/bin/env bash
set -euo pipefail

state=${STATE_DIR:-./.state}
mkdir -p "$state"

step() {
  local name=$1; shift
  if [[ -e $state/$name.done ]]; then
    echo "skip $name"
    return 0
  fi
  echo "run  $name"
  "$@"
  touch "$state/$name.done"
}

fetch()   { curl -fsSLo data.csv.partial https://example.com/data.csv; mv data.csv.partial data.csv; }
convert() { awk -F, 'NR>1{print $1}' data.csv > ids.txt.partial; mv ids.txt.partial ids.txt; }
load()    { wc -l < ids.txt; }

step fetch   fetch
step convert convert
step load    load

run  fetch
run  convert
run  load

On the second run all three lines print skip. If convert fails, set -e stops the script before its marker is written, so the next run repeats convert but not fetch. The marker goes after the command, never before, which is the whole mechanism.

Make each step safe to repeat

The step that crashed may have written half its output. Writing to name.partial and renaming with mv makes the final file appear all at once, since a rename within one filesystem is atomic. Use mkdir -p rather than mkdir, ln -sfn rather than ln -s, and INSERT ... ON CONFLICT rather than a bare insert. The pattern is expanded in atomic file writes in Bash.

Long loops need their own checkpoint

done_list=$state/processed.txt
touch "$done_list"
while IFS= read -r item; do
  grep -Fxq -- "$item" "$done_list" && continue
  process "$item"
  printf '%s\n' "$item" >> "$done_list"
done < items.txt

A step that loops over ten thousand items should record each one as it completes, so a restart picks up at the right item. For millions of items, grep per item gets slow; load the list into an associative array once, as described in associative arrays in Bash.

Reset and inspect

Add a --fresh flag that runs rm -rf "$state", and print the state directory in the help text so people know where the markers live. When the inputs change, a marker from an old run is wrong; include a hash or version of the input in the marker name, for example fetch-$(sha256sum url.txt | cut -c1-8).done.

What to record and what to recompute

A marker says "this step finished", but a step often depends on inputs that can change between runs. If the source file was replaced after the fetch step finished, the converted output is stale and the marker is wrong. Two techniques fix this. The first stores a fingerprint with the marker: write the checksum of the inputs into the marker file, and treat the step as done only when the stored value matches the current one. The second uses file timestamps the way make does, comparing the output's modification time with the input's, and rerunning the step when the input is newer. Fingerprints are more precise and cost a read of the input, and timestamps are cheap and fooled by copies that reset times. For a migration, a fingerprint of a few key parameters, such as the source database name, the date range and the script version, is usually enough to detect the common mistake of resuming a run with different arguments.

Record the script version or a hash of the script as part of the fingerprint when a fix to one step should invalidate its earlier results. Otherwise a bug fixed in the convert step leaves behind outputs produced by the buggy version, and the marker tells the next run to skip them.

Locking so two runs do not collide

Resuming makes it tempting to start the script again while the first copy is still running, for example from a second terminal. Two copies writing the same markers and files corrupt each other. Take a lock at the start with flock -n on a lock file in the state directory, and exit with a message if it cannot be taken. The kernel releases the lock when the process dies, so a crash never leaves a stale lock, which is the reason to prefer it over a pid file. Include the host name in the lock path if the state directory is on a shared filesystem, because advisory locks on some network filesystems are unreliable, and a lock held on one host may not be seen from another.

Logging that survives the interruption

A resumable script needs a record of what happened on each attempt. Append to a log file rather than overwriting it, prefix each line with a timestamp and the run id, and write a line when each step starts and when it ends. After a crash, the last lines show where it stopped and why. Send the log to the state directory so it moves with the markers. When the script finishes, print a summary of which steps ran in this attempt and which were skipped, which makes it obvious to the operator that a resume happened and did not silently redo everything.

Testing the resume path

The resume path is the one that runs only after a failure, so it needs a deliberate test. Add an environment variable such as FAIL_AT=convert that makes a step exit with an error, run the script, confirm that the markers for earlier steps exist and later ones do not, then run it again without the variable and check that only the remaining steps run and the final output is correct. Run the same test with a kill signal in the middle of a step instead of a clean error, which exercises the partial-output case. Ten minutes on this test is cheaper than discovering at the two-hour mark that the resume skips a step it should have repeated.

Cleaning up old state

Markers accumulate, and a stale directory from last month's run can make today's run skip everything. Put a run identifier or a date in the state path for jobs that are meant to start fresh each time, and keep a stable path only for jobs that are really one long task. Print the state path and the number of completed steps at the start, so an operator sees at once that the run is resuming. Delete the state when the whole job finishes successfully, after writing a final summary, so a later start is clean by default.

Where this does not apply

A step that cannot be repeated, such as sending an email or charging a card, needs its own guard, for instance an idempotency key recorded before the call. Resumability also does not help when the failure is in the data: rerunning the same bad step fails the same way. Tools that already model dependencies, such as make, give you most of this with file timestamps, see shell script versus Makefile. The GNU Make rules documentation describes the timestamp model.

Keep the state directory out of version control and out of the working tree when the script runs from a shared checkout, for example under ${XDG_STATE_HOME:-$HOME/.local/state}/jobname. That location is defined by the XDG Base Directory specification for exactly this kind of data, and it keeps concurrent runs by different users from colliding.