How to write a restore runbook you can follow at 3 a.m.
· 8 min read
On this page (8)
A backup is a pile of files. A restore is a procedure carried out by a tired human with a dead server, a customer on the phone and a half-remembered password. The gap between those two is a document, and most small projects do not have one. They have "I'll figure it out", which is a fine plan until it is 03:00 and the figuring out starts with "where did I put the repository key?".
This post gives you a template for a one-page restore runbook for a single VPS, explains how to fill in each part, translates RTO and RPO into words your customers understand, and shows how to keep the runbook true by drilling it.
RPO and RTO in plain words
Who needs this
A solo developer who is also the operations team. The risk is not that you do not know how to restore. It is that you will be stressed, will skip a step, and will have no checklist to catch the skip. A runbook is a note from calm-you to panicking-you.
A two- or three-person team where one person knows the server. If that person is on a plane, the others need the steps, the logins and the order. Write the runbook as if the reader has never seen this machine.
A freelancer running client sites. The contract probably says something vague about backups. The runbook is where you turn that into numbers you can honestly promise: how much data you could lose, how long recovery would take.
Anyone who has a good backup script and has never done a real restore. The runbook is the missing half, and writing it is the quickest way to find out which steps live only in your head.
RTO and RPO in plain language
Two terms from disaster recovery are worth keeping, once translated.
RPO, recovery point objective: how much recent work you can afford to lose. If you back up nightly, a failure at 02:55 loses almost a whole day: everything since the previous 03:15 run. So your RPO is "up to about 24 hours", and more if a backup silently failed. If that is unacceptable, the answer is a different technique, not a better runbook; for Postgres that is WAL archiving, discussed in PITR: how much can you lose.
RTO, recovery time objective: how long you can afford to be down. It runs from the moment the site breaks to the moment it works with the right data. It includes the time to notice, to decide to restore, to get a server, to install, to restore the data, to repoint DNS and to verify. Our scripted rebuild of the data and app took about half a minute in a container, which says almost nothing about a real day. Provisioning, finding credentials and DNS propagation are what actually fill the clock; the recovery drill post has the breakdown.
Write both at the top of the runbook as sentences a customer could read:
If our server is lost, we expect to be back online within about 2 hours. You could lose up to the last 24 hours of changes.
Rule of thumb: an RTO you have never measured is a wish.
Do not invent those numbers. Take them from a drill.
The template
The runbook ladder
Copy this into a file that does not live on the server. Print it. Keep one copy in your password manager's secure notes and one on paper.
RESTORE RUNBOOK - <app name> last drilled: <date> by: <name>
Measured recovery time at last drill: <hh:mm> expected data loss: up to <hh> hours
1. WHO
Incident lead (decides, talks to customers): <name, phone>
Operator (runs the commands): <name, phone>
Backup person if both are out: <name, phone>
Customer communication goes through: <status page / email list / Slack channel>
2. WHERE THINGS ARE
Server: <provider, console URL, account email>
Domain / DNS: <registrar, DNS provider, TTL now: ___ >
Backup repository: <restic repository string>
Repository password: <where it is: password manager entry NAME + paper copy in SAFE>
Offsite credentials: <where>
App secrets (env file): <where the CURRENT copy lives, not on the server>
Git remote for the app: <url>
Alerts come from: <healthcheck service / Telegram chat>
3. DECIDE
Start the clock now. Write down the time: ______
If the server is reachable, spend at most 30 minutes on diagnosis.
If not fixed in 30 minutes, or the disk or provider is gone: go to 5.
4. DIAGNOSE (stop at the first thing that is wrong)
Can you ssh in? provider console / rescue mode, provider status page
Disk or memory full? df -h / free -m
App running? systemctl status app ; journalctl -u app -n 50 --no-pager
Health URL answers? curl -i http://127.0.0.1:3000/health
Proxy up? systemctl status caddy
Bad deploy? readlink -f /var/www/app/current ; roll back the release
5. RESTORE ONTO A FRESH SERVER (in this order)
a. New Debian/Ubuntu server. Create the app user. Note the new IP: ______
b. apt install -y restic sqlite3 postgresql nodejs git curl
c. Put the repository password in /root/.restic-pass (mode 600).
export RESTIC_REPOSITORY=<string above>
d. restic snapshots --tag app --latest 3 (write down the snapshot ID and time)
e. restic restore <ID> --tag app --target / (or: latest)
f. Postgres: roles first, then the database
psql -q postgres < /var/backups/dumps/pg-globals.sql
pg_restore --create --exit-on-error -d postgres < /var/backups/dumps/pg-<db>.dump
(run as the postgres user; open the files as root and feed stdin)
g. SQLite: copy the dump to its real path, chown to the app user,
sqlite3 <file> 'PRAGMA integrity_check' must print: ok
h. Install the env file from the password manager. Deploy the app from git.
systemctl enable --now app
i. Proxy and firewall: install the Caddyfile, firewall rules.
6. VERIFY BEFORE YOU ANNOUNCE
curl -fsS http://127.0.0.1:3000/health
Row counts vs expectations: orders ______ users ______ (from the last drill)
Newest record is from: ______ (should be within the RPO)
Log in as the test user; open the record titled "______".
7. SWITCH TRAFFIC
Point the DNS A record at the new IP. Check with: dig +short <domain>
Watch the alerts and logs for 30 minutes.
8. AFTERWARDS
Run a fresh backup on the new server. Run the restore test. Write down elapsed time.
Update this runbook with anything that was wrong. Tell customers what happened.That is the one-page version. The handbook's own printable RUNBOOK.md has the same shape: a short "site is down" ladder, then the restore steps, with blanks for the server, console, password location, backup repository and alert destination.
How to fill in each part
Who
Name roles, not just people. The incident lead decides when to stop debugging and restore, and talks to customers. The operator types the commands. On a one-person team both are you, but writing the roles down reminds you that someone has to send the customer an email while the restore runs.
Put in a second person who can act, even if only to read the runbook aloud. Give them access ahead of time: a sealed note, a password manager emergency kit, a shared vault. Access arranged during an outage is access you do not have.
Where things are
This is the section that fails most often. Take each item and ask: if this server and my laptop were both gone, could I still reach it? The repository password is the critical one. Losing it means losing every backup, so it must live outside the machine it protects, in at least two places. The same goes for the current copy of your env file with API keys and database passwords, and for the offsite storage credentials.
Record the DNS TTL and lower it before you need it. A record with a 24-hour TTL turns a quick rebuild into a long tail of visitors on the dead address. Dropping it to a few minutes costs nothing during normal operation.
Decide
The most valuable line in the document is the stop-debugging rule. People who are good at debugging keep debugging long past the point where restoring would have been faster. Pick a time limit, 30 minutes is a sensible start, write it down and write down the clock start. When the time is up, the decision is made for you.
Restore
Order matters. Dependencies first, then data, then the app, then traffic. Within the data step, roles go before databases, because pg_dump -Fc does not include roles and pg_restore --create fails when the owner is missing. Put the snapshot ID and its time in the notes: the time tells you your actual RPO for this incident.
Why order matters in the data step
Make the commands copy-pasteable and explicit. "Restore the database" is not a step; the pg_restore line is. Include the gotchas you have found in drills, such as the dump directory being mode 700 so the postgres user cannot read it after a restore.
Verify
Nobody should announce recovery on the strength of an exit code. The runbook should say which counts to compare, how to read the newest record's timestamp, and one human check: log in as a test user and open a specific record. When a restore succeeds and hands you the wrong data is the failure this section exists to catch.
Switch traffic and afterwards
Move DNS last, after you have verified. Then restore your own safety net: a fresh backup on the new server, the restore test, and an update to the runbook. An incident that does not change the runbook has wasted its lesson.
Drill it, and keep it true
A runbook decays. IPs change, someone rotates a password, a package is renamed, the app gains a second database. The only defence is to run it.
- Run the whole runbook against a throwaway server, with the real person who would do it. Not the author, if you can avoid it. Their hesitations are your bug list.
- Time each section and write the total at the top. That is your measured RTO.
- Fix the document, not your memory. Every time you say "oh, you also need to..." edit the runbook before the end of the session.
- Repeat after every significant change (new database, new provider, new secret) and at least twice a year.
- Keep the cheap automated restore test running weekly between full drills; monitoring backups so silence is not success covers alerting on it.
The first drill almost always finds something: an old IP in a config file, a password in the wrong vault, a package missing from the install line.
Mistakes we see
- Storing the runbook on the server. A dead server takes it along. Keep copies elsewhere, including paper.
- Steps that say what, not how. "Restore Postgres" at 3 a.m. is a research task. The command is the step.
- No time limit on diagnosis. The longer you debug, the worse your RPO and RTO get, and the more tired the person is who has to restore.
- Promising numbers nobody has timed. An RTO written in a contract and never drilled is an obligation with no evidence.
- Secrets pasted into the runbook. Reference where they are; do not copy them in. A leaked runbook should not be a leaked key.
- Never updating after the first version. Version the document and put the last drill date in the header; a runbook last drilled two years ago is a suggestion.
Quick comparison: runbook styles
| Style | Strength | Weakness | Fits |
|---|---|---|---|
| One printed page with commands | Works with no network, easy to follow | Gets stale if not drilled | One VPS, small team |
| A script that performs the restore | Repeatable, timed | Needs a runbook for the parts it cannot do (DNS, credentials) | Anyone who can run a drill in CI |
| A wiki page | Easy to edit | On the same network or SSO as the thing that failed | Larger teams with a separate wiki host |
| In your head | Free | Fails when you are tired or away | Nobody |
The best setup is the first two together: a script that does the work and a page that tells you when to run it and what to do around it.
FAQ
What should a disaster recovery runbook for one server contain?
Who is in charge, where each credential and system lives, how to decide to restore, the restore steps in order with exact commands, verification checks, the DNS switch and a follow-up list. Keep it to a page or two so people read it under stress.
What is the difference between RTO and RPO?
RPO is how much recent data you can afford to lose, measured backward from the failure. RTO is how long you can afford to be down, measured forward. A nightly backup gives an RPO of up to a day; a measured drill gives your RTO.
How often should I test a restore runbook?
Run the cheap automated restore weekly, and the full runbook on a fresh server at least twice a year and after any large change. Update the "last drilled" date every time so you can see how stale it is.
Where should I keep the restore runbook?
Somewhere that survives losing the server and your laptop: a password manager's secure note, a git repository on another service, and a printed copy. Keep secrets out of the document and reference where they live.
Who should be able to run the runbook?
Ideally someone who did not write it. If only the author can follow it, it is a note to themselves. Give a second person access and have them run a drill.
The handbook ships a printable one-page RUNBOOK.md with a "site is down" ladder and the restore steps, plus the scripted drill that measures the time for you, in the Small App on One VPS handbook. For hardening the fresh server you restore onto, see harden a fresh Debian VPS.