How to Use a Dead Man Switch to Detect Silent Cron Failures

The worst failure of a scheduled job is the one nobody hears. The server was rebuilt without the crontab, cron stopped, the script hangs on a lock, or the disk filled before the alerting code could run. "Alert on error" cannot see any of these. The dead man's switch can, by turning the question around: expect a signal, and raise the alarm when it stops. It complements retry and alerting, and the example assumes curl 8.

The pattern

30 2 * * * /usr/local/bin/backup.sh && curl -fsS -m 10 --retry 3 -o /dev/null https://hc.example.test/ping/3f1a2b4c

The ping URL runs only if the backup exits with status 0 (the &&). The monitor knows the job should report once a day at about 02:30 and alerts if no ping arrives within the grace time. Everything that goes wrong between "cron should start it" and "it finished" produces the same visible symptom: silence. The UUID in the URL is a placeholder for the check's own identifier.

Start, success and failure signals

#!/usr/bin/env bash
ping=https://hc.example.test/ping/3f1a2b4c
curl -fsS -m 10 --retry 3 -o /dev/null "$ping/start" || true

if /usr/local/bin/backup.sh; then
  curl -fsS -m 10 --retry 3 -o /dev/null "$ping" || true
else
  rc=$?
  curl -fsS -m 10 --retry 3 -o /dev/null "$ping/$rc" || true
  exit "$rc"
fi

Hosted check services accept /start, a plain success ping and /N where N is the exit status, with non-zero meaning failure. The start signal lets the monitor measure the runtime and alert when a job runs far longer than usual, which is the hang case. Failing immediately with the code gives a faster alert than waiting for the grace period. Every ping is followed by || true so a monitoring outage never breaks the job itself.

Set the schedule and grace

Period:  1 day      expected interval between pings
Grace:   30 minutes extra time before alerting
Cron:    30 2 * * *  an alternative to a period; the monitor computes the next due time

The grace period should cover the longest normal runtime plus a margin, not the whole period. Too short and normal variance pages someone; too long and a real failure is detected late. If a job takes between ten and forty minutes, set a grace of an hour. Use the cron-expression form when the schedule is irregular, and set the monitor's time zone to match the server, a trap described in cron and daylight saving.

Self-hosted and other options

docker run -d --name healthchecks -p 127.0.0.1:8000:8000 \
  -e SECRET_KEY=change-me -e ALLOWED_HOSTS='*' \
  -e SITE_ROOT=https://hc.example.test healthchecks/healthchecks

Healthchecks is open source and can be run yourself, which avoids sending metadata to a third party, though a monitor on the same server as the jobs shares its fate; host it elsewhere. Others offer the same idea under names such as cron monitoring or heartbeat checks. A cheap do-it-yourself version is a file touched by each job and a check elsewhere that the file's modification time is recent: find /var/run/heartbeat -mmin -1500. The watcher must live on a different machine.

Why error alerts miss the worst failures

An alert that fires when a job reports an error depends on the job being alive to report it. Think of the ways a nightly task can fail without ever saying so. The scheduler is not running, because the host was rebuilt or the cron service crashed. The crontab was lost in a restore. The environment changed, and the job fails before reaching the line that sends the alert. The job starts, waits forever on a lock or a network read and never finishes. The disk is full, so the mail it tries to send cannot be queued. In every case no signal is sent, and the absence looks identical to a quiet success.

The heartbeat flips the burden. The monitor, running somewhere else, knows when each job should check in, and it is the monitor that has to be silent for things to be fine. This makes the system fail loudly in exactly the situations where the usual alerting fails silently, and it needs no knowledge of why the job stopped. You find out that the backup has not run for two days, and then you investigate the cause.

What a good check represents

Make each check represent one meaningful outcome rather than "the script started". A backup check should be sent after the archive is written and verified, a certificate renewal check after the new certificate is installed and the service reloaded. Sending the ping at the end of the real work is what gives it meaning, and sending it from a wrapper that only knows the exit status of the first command gives a weaker guarantee. If a script has several important phases, give each its own check or send progress messages with a log text, which hosted services store and show on the failing check.

Name checks after the job and the host, such as "db-backup on db1", and put the cron expression and the owner in the description. When a notification arrives at three in the morning, the name should tell the reader what is affected, and the description should say who to ask. A list of thirty checks named "backup" with identical text is useless in an incident.

Avoiding false alarms

A monitor that cries wolf gets muted, and a muted monitor is worse than none. The usual sources of noise are grace periods that are too short, jobs that vary widely in length, and maintenance windows during which jobs are intentionally stopped. Size the grace from observed runtimes, reviewing the monitor's own history of start-to-finish times to see the real spread. Pause a check during planned downtime, or better, let the maintenance script do it through the API before it starts, so no one has to remember.

Also make the alert path reliable and independent. If notifications go to a mailbox on the same server that failed, nothing arrives. Send them to at least two channels, such as chat and a mobile push service, and test the path by deliberately missing a check once a quarter. A drill takes five minutes and shows whether the alert reaches a person who knows what to do.

Checking the thing the job produced

Heartbeats confirm activity, and outputs confirm results. For a backup, add a second check that looks at the newest archive: its age, its size against the previous one, and a test restore of a small sample. Alert when the size drops by half, since that often means the job quietly backed up the wrong path. Use the heartbeat for "did it run" and the output check for "did it produce something sane", and treat them as separate failures with separate owners and separate remedies.

Where this does not apply

A heartbeat says the job ran, not that it did the right thing: a backup of an empty directory still exits 0. Put sanity checks in the job (file size, row count) and exit non-zero when they fail, as in verifying backups. A switch is also no substitute for log review. Include no secrets in the ping URL, since the identifier is the credential. See the Healthchecks documentation and the curl manual.