How to Add Retry and Alerting to Scheduled Jobs Step by Step

A job that fails once at 03:00 because a network blip stopped a download should retry, and one that fails for a week should page someone. Cron does neither on its own: it runs the command and discards the result unless mail is configured. A small wrapper supplies both, keeping the job itself simple. It works together with a dead man switch, and the script assumes Bash 5.2 and curl.

The wrapper

#!/usr/bin/env bash
# /usr/local/bin/run-job: run-job NAME COMMAND [ARGS...]
set -uo pipefail
name=$1; shift
attempts=3
delay=30
state=/var/lib/run-job/$name.fails
mkdir -p "$(dirname "$state")"
log=$(mktemp)
trap 'rm -f "$log"' EXIT

for (( i = 1; i <= attempts; i++ )); do
  if "$@" >"$log" 2>&1; then
    rm -f "$state"
    curl -fsS -m 10 --retry 3 "https://hc.example.test/ping/$name" >/dev/null || true
    exit 0
  fi
  echo "$(date -u +%FT%TZ) $name attempt $i/$attempts failed" >&2
  (( i < attempts )) && sleep $(( delay * i ))
done

fails=$(( $(cat "$state" 2>/dev/null || echo 0) + 1 ))
echo "$fails" > "$state"
if (( fails == 1 || fails % 12 == 0 )); then
  curl -fsS -m 10 -d "job $name failed on $(hostname) ($fails consecutive runs): $(tail -c 800 "$log")" \
    "https://ntfy.example.test/ops-alerts" >/dev/null || true
fi
exit 1

2026-11-03T02:30:31Z backup attempt 1/3 failed
2026-11-03T02:31:31Z backup attempt 2/3 failed
2026-11-03T02:33:01Z backup attempt 3/3 failed

Three attempts with waits of 30 and 60 seconds ride out a short outage. Output is captured to a temporary file so the alert can include the end of it. The success ping goes to a monitoring service, and the alert to a push channel such as ntfy or Slack. The hostnames are placeholders.

Use it from cron

MAILTO=""
30 2 * * * /usr/local/bin/run-job backup /usr/local/bin/backup.sh

The wrapper takes the job name and the command, so the same code serves every job. Because the wrapper exits non-zero after the last failure, cron's logs and any exit-status-aware monitoring see it. Use flock to avoid overlap; see preventing overlapping jobs.

Decide what to retry

Retry transient problems: network errors, a locked resource, a briefly unavailable API. Do not retry a failure that cannot change: a missing file, a bad configuration or an authentication error will fail three times and delay the alert by two minutes. A cheap approach is to let the job use distinct exit codes, for example 75 (EX_TEMPFAIL from sysexits) for "try again" and any other non-zero for "give up now", and have the wrapper retry only on 75. The convention for statuses is in designing exit codes.

Idempotency is the precondition

A retry reruns the whole job, so a job that half-completed must be safe to run again: write output to a temporary name and rename it, make inserts conflict-tolerant, and avoid sending an email before the step that might fail. If a job cannot be made idempotent, do not retry it automatically; alert instead. See idempotent shell scripts.

What deserves a retry

Retrying is right for failures that are likely to pass by themselves: a network timeout, a connection reset, a service returning 502 or 503, a lock held by another process. It is wrong for failures that will repeat every time, such as a syntax error, a missing file, an authentication failure or a rejected request. Retrying those wastes time, delays the alert and can cause harm, for example by locking an account after repeated bad logins. A wrapper should therefore distinguish classes of failure by exit code, and retry only the codes the job has declared transient. A single "retry everything three times" rule is easy to write and hides real problems behind delays.

Give the job a way to say what kind of failure it had. A convention that exit code 75, the sysexits value for a temporary failure, means "try again later" lets the wrapper act without parsing text.

Backoff, jitter and limits

Retries should wait, and the wait should grow. Doubling the delay after each attempt, from a few seconds up to a cap of a few minutes, gives a struggling service room to recover. Adding random jitter, so that many jobs do not all retry at the same instant, avoids a thundering herd when a shared dependency returns. Bound the total: a maximum number of attempts and a maximum elapsed time, whichever comes first, so a job scheduled every hour does not spend two hours retrying and collide with its next run.

Log each attempt with its number, the exit code and the delay before the next one. When the final attempt fails, the log should show the whole story in a few lines, which makes the alert actionable without opening other tools.

Alerting on the right thing

Alert when the job has finally failed, not on each retry. Retried successes are worth recording as a metric, because a job that needs three attempts every night is telling you about a problem even though it passes. Add a second alert for the job that did not run at all, using a heartbeat that expects a success ping within the schedule plus a margin. A crashed scheduler, a full disk and a stopped timer all produce silence, which no in-script alerting can report. Route alerts to a channel where someone will see them, include the job name, host, exit code and the last lines of output, and add a link to the runbook.

Avoid alert fatigue. Group repeated failures of the same job into one open incident, and send a recovery message when it succeeds again. Fewer, better alerts are acted on; a stream of identical ones is muted.

Overlap, timeouts and idempotency

Retries interact with locks and schedules. Take the lock before the first attempt and hold it across retries, so a second instance started by the scheduler does not run in parallel. Apply a timeout to each attempt with the timeout command, since a hung attempt otherwise never fails and never retries. And design the job so that running it twice produces the same result as running it once. A retry after a timeout may repeat work that actually finished on the other side, so a payment, an email or a deployment step needs a deduplication key or a check for existing results. Without idempotency, retries turn a transient failure into duplicated side effects, which is usually worse than the original failure.

Keep the wrapper itself small and testable. Take the command, the attempt limit, the base delay and the list of retryable exit codes as arguments or environment variables, and test the logic with a fake command that fails a chosen number of times. A wrapper that is itself unreliable moves the problem instead of solving it, and its tests are cheap to write because the behaviour is easy to describe: succeed on the third try, give up after five, never retry exit code 2.

Write down the policy for each job in its documentation: what is retried, how long it may take and who is alerted. When a job misbehaves at three in the morning, the policy is the first thing the responder needs.

Where this does not apply

For jobs with dependencies and many steps, a workflow engine handles retries, state and alerting better than a shell wrapper. Systemd can also retry a service with Restart=on-failure and notify through OnFailure=. Alert fatigue is the cost of over-alerting: one message per outage, plus a reminder, is enough. See the sysexits manual page and the systemd Restart= description.