Deploys and releases

What a plain systemctl restart costs a Go service under load

2026-11-09 · 9 min read

Most small Go services get deployed the same way: copy the new binary into place, run systemctl restart, glance at the logs. It takes two seconds and it feels free. It is not free. For the length of the restart nothing is listening on the service's port, and a reverse proxy in front of it answers every request with a 502. We wanted to know how many requests that is, so we measured it, and then measured the cheap fixes people reach for.

This post gives you the numbers, the mechanism behind them, and a script you can run against your own service in five minutes.

Who needs this

The restart cost matters differently depending on what you run. Three cases:

• A JSON API behind a mobile app. Clients are on flaky networks and some do not retry. One in four requests failing for two seconds produces a burst of error reports that points at nothing in your logs, because the app was "up" the whole time.
• A webhook endpoint. The sender retries on its own schedule. A refused connection during your restart can push the retry minutes out, and some senders count failures towards disabling your endpoint. If you receive webhooks, our post on receiving webhooks in Go covers the acknowledging side.
• An internal tool with a health probe. An uptime checker polls every 30 seconds. A 502 during a deploy pages someone, and the person on call learns to ignore the pager. This is the quiet cost: alert fatigue from your own deploys.

If your service has a handful of users who never notice two seconds, the plain restart is fine. The point of measuring is to know which case you are in.

What happens during a restart

systemctl restart app is a stop followed by a start. The order of events:

1. systemd sends SIGTERM to the process. A well-behaved Go server calls Shutdown, stops accepting, finishes in-flight requests and exits. (If yours does not, see graceful shutdown for a Go HTTP server.)
2. The process exits. The listening socket is closed with it. From this instant a connection attempt to the port is refused.
3. systemd starts the new process. It initialises, which may take milliseconds or, if it opens a database and warms a cache, a second or more.
4. The new process calls Listen. The port accepts connections again.

The gap is the time between steps 2 and 4. Caddy tries to connect to the upstream, gets "connection refused", and returns 502 Bad Gateway. The faster your service starts, the shorter the gap, but it never reaches zero because step 2 comes before step 4 by construction.

The test

A Debian 12 container with systemd 252 as PID 1, Caddy 2.6.2 in front, and a small Go service that needs about 1.5 seconds to start (an artificial initialisation delay, so the gap is easy to see). A load generator sent 20 requests per second through Caddy for the whole run, opening a new connection each time. Every fifth request went to a 300 ms endpoint. A request counted as failed if it errored, timed out, or returned anything other than 200. Each scenario ran three times, because a single run says little.

Rule of thumb: failed requests are roughly the request rate times the gap.

The result

A plain systemctl restart lost between 33 and 42 requests out of roughly 160. All of them were 502 responses from Caddy. That is about one request in four, during a restart that "only takes a couple of seconds". The command itself took between 1.6 and 2.1 seconds.

The slowest successful request in those runs took about 0.3 seconds, the same as normal. The failures were not slow, they were immediate: refused, answered with 502, done. That is why they are easy to miss in latency graphs.

Measure your own restart

Do not take our numbers for your service. Run this against a staging copy, or against production at a quiet moment with a health URL:

#!/usr/bin/env bash
# usage: ./restartprobe.sh https://example.com/health 30
url="$1"; secs="${2:-30}"
end=$((SECONDS + secs))
while [ "$SECONDS" -lt "$end" ]; do
  curl -s -o /dev/null -m 3 -w '%{http_code} %{time_total}\n' "$url" || true
  sleep 0.05
done | awk '
  { n++; if ($1 != 200) bad++; if ($2 > max) max = $2 }
  END { printf "sent=%d failed=%d slowest=%.2fs\n", n, bad, max }'

Start it in one terminal. In another, wait five seconds, then run sudo systemctl restart app. When the probe finishes you have a request count, a failure count and the slowest response. Because curl prints 000 when the connection fails, failures are counted whether they are refusals or timeouts. The loop sends a bit under 20 requests a second because of curl start-up time, which is fine for a comparison.

Run it three times. Then run it again with the changes below.

Cheap fix one: let Caddy retry

Caddy can hold a request and retry the upstream until it answers. Add lb_try_duration to the proxy:

example.com {
	reverse_proxy 127.0.0.1:3000 {
		lb_try_duration 5s
		lb_try_interval 250ms
	}
}

In our runs this brought failures to zero in all three runs of the restart scenario. It is a good setting and worth having.

The price is latency. The requests that arrived during the restart did not fail, they waited, and the slowest took 2.31 seconds, with 22 requests taking over a second. For a health check or an API call with a two-second timeout, waiting that long can be the same thing as failing. Note also that lb_try_duration only retries when the connection itself fails. A process that accepts a connection and then crashes, or a new release that answers 500 to everything, is not helped.

Cheap fix two: let systemd hold the socket

With socket activation, systemd owns the listening socket and passes it to the service. When you restart the service, the socket stays open, and connections queue in the kernel until the new process accepts them. In our test a restart of the service with a socket unit sent 161 requests and failed none.

Again the price is waiting: the slowest request took about 2.1 seconds, because it sat in the queue for the restart. When we kept two-second requests running during a deploy, the worst wait grew to 5.65 seconds. The mechanism, the code you need and the case where it is the right call are in socket activation drops nothing, but it is not free.

What actually removes the gap

Run the new version next to the old one, check that it works, and only then send it traffic. On a single server that means two instances on two ports behind the proxy, which is blue/green. In the same test with the same load, a good deploy lost zero requests and the slowest request took 0.31 seconds, the same as an idle service.

It also changes what a bad deploy looks like:

A release whose health check returned 500 never received a single real request: it was started on the idle port, failed its check, and was stopped while the old release kept serving. A release that crashed on start never got past its first second. A release that looked healthy, took the traffic and then turned sick was put back by the script with zero failed requests. In that last case the bad release did answer around 180 requests before the rollback, which is why the health check after the switch matters as much as the one before it.

The mechanics of the blue/green switch with Caddy are in the Caddy blue/green config walkthrough. The same idea with pm2 instead of systemd is in zero-downtime deploys on one VPS with Caddy and pm2.

Making the restart itself shorter

Whatever you pick, a shorter and cleaner restart helps. Three settings in the unit file are worth having:

[Service]
Type=notify
ExecStart=/opt/app/current/app
Restart=on-failure
RestartSec=1
KillSignal=SIGTERM
TimeoutStopSec=30

Type=notify makes systemctl restart return only when the process reports that it is ready, so your deploy script is not racing the start-up. That requires the app to send READY=1 once it serves, which is a small amount of Go. TimeoutStopSec bounds how long systemd waits for a graceful stop before sending SIGKILL, and it should be longer than your app's own shutdown timeout. Restart=on-failure with a short RestartSec brings the service back after a crash; pair it with StartLimitIntervalSec and StartLimitBurst in the [Unit] section so a crash loop stops instead of spinning.

Mistakes we see

• Measuring with fast requests. A restart test made of 10 ms requests can show a clean result for a service whose real requests take two seconds. Include your slow endpoints.
• Treating lb_try_duration as a deploy strategy. It hides the gap. It does not check the release, and it cannot roll back.
• Restarting from a loop. A plain systemctl restart counts toward the unit's start limit. systemd refuses to start a unit that has been started more than StartLimitBurst times (default 5) within StartLimitIntervalSec (default 10 seconds; our unit files use 60), and manual restarts count. systemctl reset-failed app clears it.
• Copying over a running binary. cp new /opt/app/current/app onto an executable that is running can fail with "Text file busy" or, with some tools, produce a half-written file. Copy to a new name and rename into place, or better, use release folders.
• A health check that returns 200 before the app is ready. Then every "healthy" report is a lie and no switch strategy can save you. See deploy health checks that mean something.
• Assuming a quiet service has no cost. The cost is per request in flight, so a service with 2-second requests loses more than a service with 20 ms requests, at the same traffic.

FAQ

Does systemctl reload avoid the downtime?

Only if your service supports it. reload runs ExecReload, commonly a SIGHUP that makes the app re-read its config without exiting. That is fine for config changes and useless for a new binary, because the process keeps running the old code.

How many requests will a restart drop on my service?

Roughly the request rate multiplied by the gap. At 20 requests per second and a gap near two seconds, that is around 40, which matches what we saw. Use the probe script above for a real answer on your own service.

Is systemctl restart or stop then start better?

They are the same sequence. restart is slightly tighter because systemd does it as one job, but the gap is the same. try-restart only restarts when the service is already running, which is handy in scripts.

Can I just make my service start faster?

You should, and it shrinks the gap, but it cannot remove it. The listener closes at step 2 and reopens at step 4, so any request arriving between them is refused. Only holding the socket (systemd) or running a second copy (blue/green) removes the refusal.

Will Caddy retries break POST requests?

lb_try_duration only retries when the connection to the upstream fails, before the request is sent, so a request that was already delivered is not repeated by this setting. Retries triggered by response status are a different, opt-in feature with its own caveats for non-idempotent methods.

Both blue/green and socket modes, plus the lock, the drain and the rollback, are what the Go + systemd Zero-Downtime Deploy Kit is made of, with the measurement tables above and the script to reproduce them.

deployssystemdgocaddyzero-downtime