Zero-downtime deploys on one VPS with Caddy and pm2

2026-11-09 · 4 min read

Most "zero-downtime deploy" posts show a config and stop there. I wanted numbers, so I put a load generator in front of a real Caddy, deployed under load, and counted failed requests. Here is the technique, the measurements, and where it does not hold.

The trap: pm2 reload does not follow your symlink

A common layout is release folders plus a current symlink:

/var/www/app/releases/20261109-101500/
/var/www/app/current -> releases/20261109-101500

You switch the symlink, run pm2 reload app, and expect the new code. I reproduced the opposite: pm2 keeps the working directory the process had when it first started. The symlink moved, the app kept running the old release, the health check passed, and a deliberately broken release was reported healthy. A deploy that silently does nothing is worse than one that fails.

Fix options: pm2 delete + pm2 start from inside current/ (works, but is an outage), or start each release from its own directory, which is what blue/green does anyway.

Blue/green on one box, no containers

Two ports, one reverse proxy.

# Caddyfile
example.com {
	encode zstd gzip
	import /etc/caddy/upstream.caddy
}
# /etc/caddy/upstream.caddy  (written by the deploy script)
reverse_proxy 127.0.0.1:3101

A deploy, simplified:

new=3102                                   # the port that is NOT serving
PORT=$new pm2 start npm --name app-b --cwd "$release" -- start
for i in $(seq 20); do curl -fsS "http://127.0.0.1:$new/health" && ok=1 && break; sleep 1; done
[ "$ok" ] || { pm2 delete app-b; exit 1; }          # unhealthy: Caddy was never touched
printf 'reverse_proxy 127.0.0.1:%s\n' "$new" > /etc/caddy/upstream.caddy.tmp
mv /etc/caddy/upstream.caddy.tmp /etc/caddy/upstream.caddy
caddy reload --config /etc/caddy/Caddyfile          # graceful: in-flight requests finish
sleep 5; pm2 delete app-a                           # drain, then stop the old one

Three details matter more than the rest:

1. Health-check the new process directly on its own port before Caddy knows about it. A broken release then never receives traffic.
2. Require several consecutive OK answers. A release that starts and dies a second later passes a single check.
3. Write the upstream file atomically and keep the old one if caddy reload fails.

What I measured

Setup: Debian 12 container, Caddy 2.6.2 from apt, pm2, a Node app that needs 1.5 s to start. Load: 20 requests per second through Caddy for the whole deploy, every fifth request a 300 ms endpoint. A request counts as failed on a connection error, a timeout over 3 s, or any status other than 200.

In the good blue/green run the load saw both versions (115 answers from v1, then 96 from v2), so traffic really moved and the zero is not an artifact of nothing happening. Counts vary a little between runs: over several runs the restart approach failed 31 to 35 requests and blue/green failed none. This is one machine and one app.

Where this does not hold

• Two releases run side by side for a few seconds, so database migrations must be backward compatible.
• Requests that last longer than your drain time after the switch can be cut (WebSockets, long uploads). I did not test those.
• One process per slot, no cluster mode.
• I tested Caddy 2.6.2 only. A failing caddy reload path and systemd-managed Caddy were not tested.

The boring half: can you restore?

A deploy script is half the story. The other half is a backup you have actually restored. My rule: dump SQLite with .backup (never copy a live file), dump Postgres with pg_dump -Fc, abort the whole run if a dump is corrupt, and schedule a restore test that restores the latest snapshot to a temp directory and checks the dumps. In my test a corrupted SQLite file stopped the backup without creating a snapshot, which is exactly the behaviour you want.

I packaged the scripts (deploy with both modes plus rollback, restic backup with database dumps, a restore test, failure alerts to Telegram or a webhook) as the VPS Deploy & Backup Kit, $19, with the test suite results in the README, including what I did not test. The technique above needs none of it; the snippets are enough to build your own.