I destroyed my server on purpose and timed the recovery

2026-11-09 · 2 min read

"We have backups" is a sentence, not a result. The result is a number: how long from a dead server to a working app with the data intact. So I scripted the whole thing and timed it.

The drill

Source "server": a clean Debian 12 container with a small Node app, a SQLite database, a Postgres database and a folder of uploads, backed up with restic to a volume that lives outside the server. Then:

Destroy the source container. Only the backup volume remains.
Start a fresh debian:bookworm.
Install packages (restic, Postgres, Node, git).
restic restore, restore the Postgres roles and databases, copy SQLite back and integrity-check it.
Deploy the app from git and wait for its health endpoint.
Verify row counts and a checksum over every upload.

Measured, on a laptop under Docker Desktop (arm64):

	Small data	Larger data
Data	100k rows per database, 200 x 100 KB uploads	2M rows per database, 500 x 1 MB uploads
Total recovery	29.8 s	34.3 s
`apt` install	24.1 s	23.0 s
`restic restore`	1.6 s	3.3 s
`pg_restore`	2.6 s	4.9 s

The package install dominated and varied a lot (21 s to 57 s across runs), so totals ranged from about 27 s to 62 s. Provisioning a real server at a provider is not in these numbers, and neither is your DNS change. Treat this as "the scripted steps are cheap; the part that bites is everything around them".

Five bugs the drill found in my own scripts

None of these would have shown up in a "backup succeeded" log line.

Roles are not in pg_dump -Fc. Restoring into a fresh cluster failed with role "shop" does not exist. You need pg_dumpall --globals-only as well.
Permissions on the dump directory. I keep dumps in a mode-700 directory (good). After restic restore the postgres user could not read the dump (also correct, and exactly the kind of thing you want to meet on a calm Tuesday).
Two deploys in the same second shared a release folder, and the cleanup after the failed second deploy deleted the live release.
A crash-looping release hit systemd's start limit, so the rollback's restart was refused until systemctl reset-failed.
The drill ignored my data-size settings. docker exec does not forward the host environment, so the "large" run was secretly the small one until I printed the counts.

What I'd copy

Put the restore in a script, run it weekly from cron, and make it fail loudly. A restore you do by hand once a year will not work the year you need it.
Verify with counts and checksums, not "the command exited 0".
Time it. The number is what you tell your users.
Write down what you did NOT verify. Mine: real certificate issuance, a real reboot with swap, Ubuntu-specific behaviour, remote restic backends, point-in-time recovery for Postgres.

The scripts, the drill, config templates and 15 short chapters are in the "Small App on One VPS" handbook ($29). It has a status table that says what was run and what was not, and one command that re-runs every check in Docker.