Backups and recovery

Monitoring backups so that silence never means success

2026-11-09 · 9 min read

Most backup monitoring is an alert on failure, and it has a blind spot the size of the whole problem. A script that fails can tell you it failed. A script that never started tells you nothing. The cron daemon stopped, the crontab was overwritten during a rebuild, the server was off, the disk was read-only, the script lost its execute bit in a deploy. Every one of those looks, from the inbox, exactly like a quiet night with a healthy backup.

This post builds monitoring in layers: alerts for failures, a dead-man's switch for absence, an independent check of what is actually in the repository, and a habit of breaking things on purpose to prove the alarms ring.

Who needs this

A developer with a nightly cron job and a log file. The log says backup OK every night until the night cron stops, and then it says nothing. Nobody reads a log that is not growing. The change: something outside the box expects a ping each day and shouts when it does not arrive.

A small team that rebuilt the server last month. Rebuilds reset crontabs and systemd timers. The new server runs the app and nothing else, and the backup job was on the old one. A dead-man's switch would have flagged it the next morning.

Someone whose alert goes to email on the same server. MAILTO in cron sends to a local mailbox on the machine you are monitoring. When the machine is down you cannot read it, and when it is up you probably never look. The change: alerts that leave the server for a channel you actually read.

An operator whose backup "succeeds" while backing up nothing. The job exits 0 and the snapshot is three kilobytes, because the dump step quietly produced an empty file. Exit codes were green. A size and age check on the result catches it.

Four ways a backup goes wrong, and which signal catches each

No single column covers every row. You need the first two for each run and the last two as an independent view of the result.

Layer 1: failure alerts

The script that does the work should say loudly when it fails. In a bash backup job this is an EXIT trap that reports a non-zero status:

set -euo pipefail
NOTIFY_SCRIPT=backup.sh
trap 'notify_exit $?' EXIT

The VPS Deploy & Backup Kit's notify.sh implements it as a small library that backup.sh and restore-test.sh source. It sends a message only when the script exits non-zero, through a Telegram bot, a webhook, or both. Two details matter when you write your own. Pass the bot token to curl on standard input (curl -K -) rather than on the command line, so it does not show in ps output. And make a failed notification never change the script's own exit status, so an unreachable Telegram does not turn a good backup into a failed one.

Failure alerts have a limit worth stating plainly. They run inside the script. If the script is killed with SIGKILL, never starts, or the host is down, no trap fires. That is the case for layer 2.

Layer 2: a dead-man's switch

Invert the logic. Instead of the job reporting failure, an external service expects a success ping on a schedule and alerts when the ping is late. Hosted services in the Healthchecks style (and the open-source Healthchecks you can run yourself) work like this: you create a check with a period, such as one day, and a grace time, such as two hours, and you get a unique ping URL. If no ping arrives within period plus grace, the service sends the alert.

At the end of a successful backup:

curl -fsS -m 10 --retry 3 -o /dev/null "$BACKUP_HEALTHCHECK_URL" || true

-fsS makes curl fail on HTTP errors but print them, -m 10 bounds the wait, --retry 3 rides out a blip, and || true ensures the ping can never break the backup. The ping goes only after everything succeeded: dumps verified, snapshot written, forget done, check passed. The kit does exactly this at the end of backup.sh, using a URL you put in kit.env.

Most of these services accept more than a bare ping, and it is worth using. A /start signal before the job and a final signal after lets the service measure run time and notice a job that started and never finished. A /fail signal reports failure immediately instead of waiting for the grace time to expire. Some also take the exit status as a path suffix. A wrapper that uses all three looks like this:

#!/usr/bin/env bash
URL="https://hc-ping.example.com/your-check-uuid"
curl -fsS -m 10 -o /dev/null "$URL/start" || true
/opt/backup/backup.sh
rc=$?
curl -fsS -m 10 -o /dev/null "$URL/$rc" || true
exit "$rc"

Check your service's documentation for the exact suffixes; the idea is the same everywhere. Give the restore test its own, separate check with a period of a week. The kit supports this with a second URL, RESTORE_HEALTHCHECK_URL, and sends the /fail variant when the restore test fails.

Choose period and grace on purpose. If the job runs at 03:15 and takes ten minutes, a one-day period with a two-hour grace fires in the morning, early enough to act.

• A grace of a day means you find out two days late.
• A grace of five minutes means you are woken by every slow night.

Rule of thumb: the grace time is the longest you are willing to be blind.

Layer 3: ask the repository how old the newest snapshot is

Pings tell you the script finished. They cannot tell you the repository holds a fresh, plausible snapshot. Ask it directly, ideally from a different machine, so the check does not depend on the thing it checks:

#!/usr/bin/env bash
set -euo pipefail
export RESTIC_REPOSITORY RESTIC_PASSWORD_FILE
MAX_AGE_H=${MAX_AGE_H:-26}
latest=$(restic snapshots --tag app --json | jq -r 'map(.time) | max // empty')
[ -n "$latest" ] || { echo "no snapshots found"; exit 1; }
age_h=$(( ( $(date +%s) - $(date -d "$latest" +%s) ) / 3600 ))
[ "$age_h" -le "$MAX_AGE_H" ] || { echo "newest snapshot is ${age_h}h old (limit ${MAX_AGE_H}h)"; exit 1; }
echo "newest snapshot is ${age_h}h old"

The limit is 26 hours for a nightly job: a full day plus slack. The script uses GNU date -d, which Debian and Ubuntu have. Wire it up as another cron job on the monitoring host, and give it a dead-man's switch of its own: a monitor that silently stops is the same problem one level up.

This also covers the failure mode where the backup job is running and pinging happily against the wrong repository, or where a repository path changed and the nightly snapshot lands somewhere you are not looking. It reads the same repository you would restore from.

Layer 4: check what is in the snapshot

A green job can still produce a useless snapshot. Cheap checks inside the backup script catch the worst of it:

# refuse a dump that has shrunk to less than half of yesterday's
size=$(stat -c %s "$DUMP_DIR/pg-appdb.dump")
prev=$(cat /var/lib/backup/last-size 2>/dev/null || echo 0)
[ "$size" -ge $(( prev / 2 )) ] || { echo "pg-appdb.dump is $size bytes, was $prev"; exit 1; }
echo "$size" > /var/lib/backup/last-size

Halving is arbitrary, so pick a threshold from your data. A shrinking dump is a warning sign, not proof of anything, which is why this check should alert rather than silently proceed. The weekly restore test goes further: it restores the latest snapshot, checks every dump and counts rows. What those counts should be compared against is the topic of when a restore succeeds and hands you the wrong data.

Exit codes: what restic tells you

Do not collapse everything to pass or fail without knowing what the tool says. restic exits 0 on success and 1 on a fatal error. For restic backup, exit code 3 means the snapshot was created but some source files could not be read, for example because they vanished or were unreadable mid-run. Newer versions add distinct codes for a missing repository, a locked repository and a wrong password; read the exit-code section of the manual for the version you run.

Decide per case. A strict job that treats exit 3 as failure is a reasonable default for a server whose data should always be readable, and it is what set -e gives you automatically. If your job legitimately sees files disappear (temp files in a watched folder), handle code 3 explicitly rather than ignoring all non-zero statuses. Never write restic backup ... || true.

And always use set -euo pipefail in the wrapper. Without pipefail, pg_dump | gzip > out reports gzip's success even when the dump died.

With systemd instead of cron

A timer gives you two things cron does not: Persistent=true runs a job that was missed while the machine was off, and OnFailure= can trigger an alert unit.

# /etc/systemd/system/backup.service
[Unit]
Description=Nightly backup
OnFailure=backup-alert.service

[Service]
Type=oneshot
ExecStart=/opt/backup/backup.sh

# /etc/systemd/system/backup.timer
[Unit]
Description=Nightly backup timer

[Timer]
OnCalendar=*-*-* 03:15:00
Persistent=true

[Install]
WantedBy=timers.target

# /etc/systemd/system/backup-alert.service
[Unit]
Description=Backup failure alert

[Service]
Type=oneshot
ExecStart=/opt/backup/alert.sh backup.service failed

Enable with systemctl enable --now backup.timer, and see what is scheduled with systemctl list-timers. OnFailure= still cannot fire if the whole host is down or the timer is not enabled, so you keep the dead-man's switch either way.

Prove the alarms work

Monitoring that has never fired is a guess, in the same way a backup you never restored is. Once a quarter, break things on purpose and watch the alerts arrive.

1. Point RESTIC_REPOSITORY at a wrong path and run the job. The failure alert should arrive within a minute.
2. Disable the cron line or timer for a day. The dead-man's switch should alert after period plus grace.
3. Fill the dump directory's filesystem in a scratch environment, or make a dump fail with a bad database name. The job should abort before the snapshot.
4. Block the notification host in the firewall for one run. The job should still exit with its real status.

Write down how long each took to alert. If the number is "a day and a half", you have found out what your grace times mean.

Mistakes we see

• Alerts that go to the machine being monitored. Local mail and local log files are readable only when things are fine.
• Pinging at the start of the script, or in a trap that also runs on failure. The success ping belongs after the last check passed.
• One check for everything. Give the backup, the restore test and the age check separate dead-man's switches with their own periods.
• A grace period longer than your tolerance. If you cannot lose more than a day, a two-day grace is a bug.
• Ignoring exit code 3 with || true. You will back up a partial tree and not know.
• Treating the monitor as done once. Hooks, tokens and URLs rot. Rotate and retest them.

FAQ

What is a dead-man's switch in backup monitoring?

It is an alert that fires when an expected signal stops arriving. Your backup job pings a URL after each successful run, and the monitoring service alerts when a ping is late. It catches the failures a script cannot report about itself, such as cron not running or the server being down.

How do I know my backup cron job actually ran?

Do not rely on the log; a job that never ran writes nothing. Use a dead-man's switch for presence, and an independent check of the newest snapshot's age for results. With systemd timers, systemctl list-timers and journalctl -u backup.service show runs.

Should the alert come from the script or from the monitoring service?

Both. The script alerts with detail when it fails, for example "SQLite dump is corrupt". The service alerts when the script cannot speak at all. The detail helps you fix it, the silence detector keeps you from missing it.

How do I check the age of the latest restic snapshot?

Run restic snapshots --json, take the newest time field and compare it with the current time, as the script above does. Run it from another machine and alert when the age exceeds your schedule plus a margin.

Can I monitor backups without a third-party service?

Yes. Run the age check from a second server on a cron schedule and send its own alert, or self-host an open-source Healthchecks instance. The principle that matters is that the watcher lives somewhere other than the thing it watches.

The VPS Deploy & Backup Kit wires layers one and two in already: failure alerts through Telegram or a webhook, plus separate healthcheck URLs for the backup and for the weekly restore test. The patterns above are enough to add the other two layers yourself. For the wider picture of watching one server, see monitoring one VPS without a stack.

backupmonitoringresticcron