Deploys and releases

Caddy blue/green config for a one-server app, line by line

2026-11-09 · 10 min read

You have one server, one app, and a reverse proxy in front of it. You want to ship a new version without a window where visitors get a 502. The cheapest way to do that without containers is to run two copies of the app on two ports and let Caddy decide which one gets traffic. This post walks through the Caddy side of that setup: the config, what a reload really does, which health check options matter, and where the design bites.

Who needs this

• A Go or Node API behind Caddy that restarts on every deploy. Each restart is a few seconds of refused connections. Mobile clients retry, browsers show an error page, a webhook sender marks you as failing. Two upstream ports turn the restart into a switch.
• A small SaaS with a login-protected dashboard. Users hold sessions and long-ish requests (exports, uploads). You want a deploy to finish while a 20 second export is still running, not cut it.
• A solo developer who deploys during the day. If a deploy is scary you batch changes and ship at night. When a bad release never receives traffic, you can ship one small change at a time.
• An agency box with several apps. One Caddyfile, one small import per app, and a deploy script that touches only its own snippet.

If your app already runs several replicas behind a real load balancer, you have this solved elsewhere. This is for the single-VPS case. The broader argument for it is in zero-downtime deploys on one VPS; the cost of skipping it is measured in what a plain systemctl restart costs under load.

The shape of the setup

Two instances of the same app listen on 127.0.0.1:3101 and 127.0.0.1:3102. At any moment one of them is live. Caddy proxies to the live one. A deploy starts the release on the idle port, checks that it works, then tells Caddy to switch. The old instance drains and stops.

The Caddyfile never changes between deploys. It imports a tiny file that does:

example.com {
	encode zstd gzip
	import /etc/app/upstream.caddy
	log {
		output file /var/log/caddy/example.com.log
	}
}

And the imported file, written by the deploy script, is one line:

# generated by deploy.sh, do not edit by hand
reverse_proxy 127.0.0.1:3101

This is the layout the Go + systemd deploy kit uses in its default mode. Its install.sh creates the snippet once, pointing at the first port, and deploy.sh rewrites it.

Why a separate file and not an edit of the main Caddyfile? Because the deploy script should own exactly one tiny, fully generated file. If it did sed on your real Caddyfile, a stray comment or a second site block would eventually confuse it. With import, the main file stays human, the snippet stays machine, and the script can overwrite the snippet without parsing anything.

Writing the snippet safely

Two details in the script matter more than they look.

First, the write is atomic. The script writes upstream.caddy.tmp and then renames it over the real file. A reload that happens at the wrong instant never reads a half-written file, because mv within one filesystem is a single rename.

write_snippet() {
  printf '# generated by deploy.sh, do not edit by hand\nreverse_proxy 127.0.0.1:%s\n' "$1" >"$CADDY_SNIPPET.tmp"
  chmod 0644 "$CADDY_SNIPPET.tmp"
  mv -f "$CADDY_SNIPPET.tmp" "$CADDY_SNIPPET"
}

Second, the chmod 0644. Caddy runs as its own user. A script running as root under umask 077 produces a file Caddy cannot read, the reload fails, and you spend ten minutes staring at a permissions error that has nothing to do with your release. Set the mode explicitly.

What systemctl reload caddy actually does

On Debian and Ubuntu the packaged unit defines the reload as caddy reload --config /etc/caddy/Caddyfile --force. That command does not restart the process. It loads the Caddyfile, adapts it to Caddy's JSON config, validates it, and posts it to the admin API on localhost:2019. Caddy then starts the new configuration and retires the old one.

Three properties make this safe for our purpose:

1. A config that fails validation is never applied. If the snippet has a typo, the reload returns an error and Caddy keeps serving the old config. Your deploy script sees a non-zero exit and can put the old snippet back.
2. Listening sockets are reused. Port 443 is not closed and reopened, so a client connecting during the reload is not refused.
3. In-flight requests finish on the old config. A request that was already proxied to :3101 keeps its connection to :3101. New requests go to :3102.

That third point is why the old instance must not be killed the moment the snippet changes. Requests that started before the switch are still running on it. The kit waits DRAIN_SECONDS (default 3) and then runs systemctl stop on the old unit, which is itself graceful: the app receives SIGTERM and finishes what it is doing. If your longest normal request is 20 seconds, raise the drain value. Caddy also has a global option for how long it waits for connections to close when a config is replaced, grace_period, which is worth knowing about if you see long requests being cut at reload time. Set it in the global options block at the top of the Caddyfile.

Rule of thumb: switch first, stop the old instance last. Requests already in flight finish on the instance that started them.

You can try all of this by hand:

caddy validate --config /etc/caddy/Caddyfile
printf 'reverse_proxy 127.0.0.1:3102\n' | sudo tee /etc/app/upstream.caddy
sudo systemctl reload caddy
curl -sI https://example.com/ | head -1

caddy validate is the cheap guard. Put it before the reload in any script of your own.

Order of operations in a deploy

The Caddy switch is one step of several, and the order is the whole point:

1. Start the new release on the idle port.
2. Health-check that port directly, not through Caddy.
3. Write the snippet and reload Caddy.
4. Check again after the switch.
5. Wait for the drain, stop the old instance.

Step 2 happens on 127.0.0.1:3102, bypassing the proxy. That matters. If you check through Caddy before the switch, you are checking the old release. Checking the new one means talking to its port. The post on deploy health checks that mean something covers what that endpoint should verify.

Step 3 has a failure branch the kit handles: if the reload fails, the previous snippet is written back, Caddy is reloaded to it, the new instance is stopped, and the old one was never touched. Step 4 has another: if the new release passes before the switch but fails right after (a lazy connection pool, a cache warm-up that crashes), the script switches Caddy back to the old instance, which is still running, and exits non-zero. This is the main reason to keep the old instance alive until the post-switch check passes.

Health checks inside Caddy

Everything above checks health from outside Caddy, in the deploy script. Caddy has its own health checking, and it is worth being clear about what it is for.

Active health checks poll an upstream and take it out of rotation if it fails:

reverse_proxy 127.0.0.1:3101 127.0.0.1:3102 {
	lb_policy first
	health_uri /health
	health_interval 2s
	health_timeout 1s
	health_status 2xx
	lb_try_duration 3s
	lb_try_interval 250ms
}

With lb_policy first, Caddy sends everything to the first healthy upstream in the list, so 3102 only takes traffic when 3101 is down. This is a different way to do blue/green: no file rewrite, no reload, Caddy fails over by itself. It is good as a safety net, and some people run it as their whole strategy.

We still prefer the explicit switch for deploys, for two reasons. A deploy is a decision, and you want it recorded (a state file, a log line, a symlink), not inferred from whichever port happens to answer. And two upstreams in one list means the new release and the old one are both candidates during every interval, so you cannot say which one served a given request.

The two ideas combine well, though. Passive and active settings on the single live upstream help during the drain window:

• lb_try_duration and lb_try_interval make Caddy retry a failed connection for a short time instead of returning 502 at once. If the old instance stops a moment before an in-flight keep-alive is retired, the retry covers the gap.
• health_uri on the single upstream is optional, but it gives you Caddy's own view in the admin API, which is handy when debugging.

Keep the retry window short. Retrying for 30 seconds hides an outage instead of absorbing a hiccup.

Comparing the options

Mistakes we see

Checking health through Caddy before the switch. The check passes because the old release answers. Always check the idle port directly.

Editing the live Caddyfile from the script. One bad sed and every site on the box is affected. Use import and a generated file.

Forgetting that the app must tolerate two versions at once. For a few seconds both releases run. If release N+1 changes the database in a way release N cannot read, those seconds are an outage. See migrations during zero-downtime deploys.

Binding the app to 0.0.0.0. The instances should listen on 127.0.0.1 only. Caddy is the one door. Otherwise port 3102 is reachable from the internet and is a second, unproxied copy of your app.

Reloading with systemctl restart caddy. A restart drops the listeners. Use reload. In scripts, make the command configurable (the kit has CADDY_RELOAD_CMD) so that you can swap in caddy reload if you run Caddy outside systemd.

A drain shorter than your slowest request. The old process gets SIGTERM, your app has a 25 second shutdown limit, but the script moved on and a different timeout kills it. Keep the app's shutdown timeout below systemd's TimeoutStopSec, and the drain wait at least as long as the requests you care about.

Never testing the reload failure path. Break the snippet on purpose in a staging box (write reverse_proxy with no argument) and confirm that Caddy keeps serving and your script restores the old file. You want to see that once before a real deploy does it.

FAQ

Does caddy reload drop connections?

No. Caddy keeps the listening sockets across a reload, validates the new config before applying it, and lets requests that are already in flight finish. A request that is mid-way to the old upstream stays on it. What can still hurt you is stopping the old upstream too early, so give it a drain period.

Do I need a reload at all, or can Caddy discover the new port itself?

You can skip the reload by listing both ports as upstreams with lb_policy first and active health checks, and Caddy will fail over on its own. You lose the explicit record of which release is live and you get overlap you cannot attribute. For deploys we prefer a deliberate switch and keep Caddy's health checks as a safety net.

Where should the upstream snippet live?

Outside /etc/caddy, in a directory owned by your app, for example /etc/app/upstream.caddy. Caddy needs read access, the deploy script needs write access, and neither should be editing the main Caddyfile. Make the file world-readable (mode 0644) because Caddy runs as an unprivileged user.

What happens if the reload fails halfway through a deploy?

Caddy refuses invalid configs and keeps the old one, so traffic keeps flowing to the old instance. A good deploy script then restores the previous snippet, stops the new instance and exits with an error. The kit does exactly this, so a failed reload costs you nothing but a failed deploy message.

Can I use the same pattern with nginx or HAProxy?

Yes. The shape is identical: a generated upstream file, a validated graceful reload (nginx -t && nginx -s reload, or HAProxy's reload), and a health check against the idle port first. Caddy makes it a little shorter because import and automatic HTTPS remove two other files you would otherwise manage.

The Go + systemd deploy kit packages this exact flow, with the snippet handling, the reload fallback, the post-switch check and the automatic rollback already written and exercised against a real Caddy.

caddydeploysystemdzero-downtime