How to Set Up Basic Alerting Without a Big Platform

A three-person team does not need a monitoring platform to know that the disk is full. It needs a handful of checks, a channel everyone reads, and a way to find out when the checker itself has died. This post builds that with shell and tools you already have. It is the lightweight start before Prometheus and node exporter. The commands assume Debian 13 with ntfy as the push channel.

The checks

#!/usr/bin/env bash
set -u
state=/var/lib/checks
mkdir -p "$state"
host=$(hostname -s)

alert() {   # name, message
  local f="$state/$1"
  if [ ! -e "$f" ]; then
    : > "$f"
    curl -fsS -H "Title: $host: $1" -H "Priority: high" -d "$2" https://ntfy.example.org/ops >/dev/null
  fi
}
clear_alert() {
  local f="$state/$1"
  if [ -e "$f" ]; then
    rm -f "$f"
    curl -fsS -H "Title: $host: $1 recovered" -d "back to normal" https://ntfy.example.org/ops >/dev/null
  fi
}

use=$(df --output=pcent / | tail -n1 | tr -dc '0-9')
if [ "$use" -ge 85 ]; then alert disk "root filesystem at ${use}%"; else clear_alert disk; fi

failed=$(systemctl --failed --no-legend | wc -l)
if [ "$failed" -gt 0 ]; then alert units "$failed failed systemd units"; else clear_alert units; fi

if curl -fsS --max-time 10 https://example.org/health >/dev/null; then clear_alert health; else alert health "health check failing"; fi

curl -fsS --max-time 10 https://hc-ping.com/YOUR-UUID >/dev/null

The state file under /var/lib/checks is what prevents repeats: an alert is sent when a check first fails, and a recovery message when it passes again, not every five minutes in between. Without this, a disk that stays full sends 288 messages a day and people learn to mute the channel. The final curl is the heartbeat; see below.

Schedule it

*/5 * * * * root /usr/local/bin/checks.sh

A systemd timer works equally well and logs to the journal, as described in cron versus systemd. The script exits 0 whatever happens, since a failing check is data, not an error of the checker.

The heartbeat

A monitor that runs on the monitored host dies with it. A heartbeat service such as Healthchecks.io expects a ping every few minutes and sends its own alert when pings stop. That covers the host being down, cron stopping or the network failing, the cases where your own script cannot speak. See dead man switch monitoring. Run one external check of the public URL from a second machine or a hosted uptime service as well, since a local check cannot see what a user on the internet sees.

What to alert on

• Page-worthy: the site is down, the disk will fill within hours, backups have not run, a certificate expires in under 7 days.
• Ticket-worthy: a single failed unit that restarted, a slow trend in disk use.
• Log-only: CPU above 80 percent for a minute, a one-off error.

If an alert has no action attached, remove it. Each alert should map to a line in a runbook; see writing a runbook.

Where this does not apply

Scripts do not give history, graphs or per-service rules, and they get unwieldy past a dozen hosts. At that point use a metrics system. Do not alert by email alone to a mailbox nobody watches, and test the path end to end by lowering a threshold on purpose: an alert that has never fired is unproven. Messages go through ntfy or Slack webhooks.

What a small team actually needs to know

Alerting is a decision about which problems deserve to interrupt a person. A small team does not need hundreds of signals. It needs to hear about the handful of failures that cost money or trust: the site is down, the disk is nearly full, the backup did not run, the certificate is about to expire, and the machine has stopped reporting at all. Everything else goes to a dashboard or a daily summary that someone reads over coffee. The test for an alert is simple. If it fires at 3 a.m., is there an action a person can take that cannot wait until morning? If not, it is a report, not an alert. Teams that skip this step end up with so many alerts that all of them get muted, and the first real outage hides inside the noise.

Start from symptoms that users feel, such as failed requests and slow pages, and add causes such as full disks only when they predict a failure with enough lead time to act. A disk at 85 percent that grows a percent a week is a ticket. A disk at 97 percent that grows five percent an hour is a page.

Layers: inside checks, outside checks and the silence check

A check that runs on the machine it watches fails with that machine. If the host loses power or the network, the script that would have reported the problem is gone with it. So combine three layers. Local checks run on each server and report disk, memory, services and jobs, which catches slow failures early. An outside probe runs from another network, such as a small virtual machine or a hosted monitor, and requests the public URL, which catches what users see, including DNS, certificates and the load balancer. The third layer is the silence check, often called a heartbeat or dead-man switch: each job pings a URL when it finishes, and a service raises an alarm when the ping does not arrive on time. It catches the failure mode that nothing else can, a check that quietly stopped running.

Keep each layer simple enough to trust. A monitoring setup you cannot understand during an outage will not be fixed during one either.

Choosing where messages go and what they say

Send alerts somewhere a person will notice at the right urgency. A push notification or a phone call for the pages, a chat channel for warnings, and email for reports. Test the path end to end, including your phone's do-not-disturb rules, since a perfect alert that never reaches anyone is a failure. Write the message so that it can be acted on from a lock screen: the host, what is wrong, the current value and the threshold, and a link to the runbook step. "Disk /var on db1 is at 96 percent, threshold 90, runbook link" is a message. "Alert triggered" is not. Add a rate limit or a repeat interval so a persistent problem sends a reminder every hour and not every minute.

Agree who is on call, even if the team has three people. An alert that goes to a shared channel with no owner is read by everyone and answered by nobody. A rota, however simple, turns "someone should look at that" into a name.

Review and tune every month

An alerting setup drifts. Once a month, look at every alert that fired and ask whether it led to an action. Remove or downgrade the ones that did not, and add a check for any incident that was found by a customer before it was found by you. Keep a list of the alerts with the owner, the threshold and the reason it exists, which also tells new team members what the noise means. Count the pages per week as a health measure for the team: a rising number is a signal to fix causes and not to get used to being woken up.

Plan for the day the monitoring itself breaks. Keep one cheap external check that does not share infrastructure with the rest, and make sure at least two people can reach the alerting configuration and know how it works.

See the ntfy publishing documentation and Healthchecks.io docs.