Deploys and releases

systemd socket activation: no dropped requests, but a cost

2026-11-09 · 9 min read

Search for how to restart a service on systemd without dropping connections and socket activation is the answer you will keep finding. It is a good answer. systemd opens the listening socket, hands it to your process, and keeps holding it while the process restarts, so clients are never refused. We tested it carefully with a Go service behind Caddy, and it did exactly that. It also came with a cost that most write-ups skip: requests do not fail, they wait, and a request that waits long enough is a failed request to a client with a short timeout.

This post explains how socket activation works, shows the unit files and the Go code, gives the numbers we measured, and says when we would choose it.

Who needs this

Socket activation earns its place in a narrower set of situations than the blog posts suggest. Three where it fits well:

• A service you cannot run twice. It holds an exclusive lock on a file, owns a serial device, or listens on a fixed port that something outside your control points at. Blue/green needs two copies at once, so it is off the table, and socket activation is the next best way to avoid refusals.
• A small internal API with short requests. A few dozen requests per second, each finishing in tens of milliseconds, called by scripts that retry. A restart that queues connections for two seconds is invisible to them.
• A box with no reverse proxy. The service listens on its own port and has no Caddy to add retries in front of it. The socket unit gives you the "do not refuse" behaviour at the OS level, with no other software involved.

Two situations where it fits badly: services with long requests (uploads, reports, anything that takes seconds), and services whose clients have short timeouts. We come back to both below.

What it does

Normally your service opens its own listening socket. When the process exits, the socket closes, and anything that connects before the next process calls Listen is refused. That window is the gap described in what a plain systemctl restart costs under load.

With socket activation, a .socket unit makes systemd create the listening socket. When a connection arrives, or when you start the service, systemd starts the matching .service and passes the already-open socket as file descriptor 3, with the environment variables LISTEN_PID and LISTEN_FDS telling the process what it received. When you restart the service, systemd keeps the socket. The kernel completes the TCP handshake for new connections and holds them in the socket's backlog queue. When the new process starts accepting, it finds them waiting.

Nothing is refused, because the socket never closed. That is the whole trick.

Rule of thumb: restart app.service, never app.socket.

The unit files

Two files. The socket:

# /etc/systemd/system/app.socket
[Unit]
Description=app listening socket

[Socket]
ListenStream=127.0.0.1:3000
Backlog=1024
NoDelay=true

[Install]
WantedBy=sockets.target

And the service, with the hardening trimmed for readability:

# /etc/systemd/system/app.service
[Unit]
Description=app (socket activated)
Requires=app.socket
After=app.socket network.target
StartLimitIntervalSec=60
StartLimitBurst=5

[Service]
Type=notify
NotifyAccess=main
User=app
Group=app
WorkingDirectory=/opt/app/current
EnvironmentFile=/etc/app/app.env
ExecStart=/opt/app/current/app
KillSignal=SIGTERM
TimeoutStopSec=30
Restart=on-failure
RestartSec=1
NoNewPrivileges=yes
ProtectSystem=strict
PrivateTmp=yes

systemd pairs app.socket with app.service by name. Enable the socket, not the service:

sudo systemctl daemon-reload
sudo systemctl enable --now app.socket
sudo systemctl status app.socket app.service
ss -ltnp 'sport = :3000'

The ss output shows systemd (PID 1) as the owner of the port. After the first connection, app.service is running and holds the same socket as well. A restart of app.service leaves app.socket active. Do not systemctl restart app.socket during a deploy: that closes and reopens the socket and brings the gap back.

Backlog= sets how many pending connections the kernel will queue. Under restart load the queue is what absorbs your traffic, so give it room. The kernel also caps the value (net.core.somaxconn), so a number far above that limit is cut down.

The code in the app

The app has to accept the inherited socket instead of opening its own. In Go the standard library is enough:

func listener(addr string) (net.Listener, error) {
	if os.Getenv("LISTEN_PID") == strconv.Itoa(os.Getpid()) &&
		os.Getenv("LISTEN_FDS") == "1" {
		f := os.NewFile(3, "systemd-socket")
		defer f.Close() // FileListener dups the descriptor
		return net.FileListener(f)
	}
	return net.Listen("tcp", addr) // not socket-activated: listen ourselves
}

The LISTEN_PID check matters. systemd sets the variables for the process it starts, and a child that inherits the environment must not claim a descriptor that belongs to its parent. Falling back to a normal net.Listen keeps go run working on your laptop. If you would rather not maintain this yourself, the coreos/go-systemd module has an activation package that does the same job.

Then serve and shut down properly:

ln, err := listener(":3000")
if err != nil { log.Fatal(err) }
srv := &http.Server{Handler: mux, ReadHeaderTimeout: 5 * time.Second}

go func() {
	if err := srv.Serve(ln); err != http.ErrServerClosed { log.Fatal(err) }
}()
sd.Notify("READY=1") // writes the state string to the NOTIFY_SOCKET unix datagram socket

ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM, os.Interrupt)
defer stop()
<-ctx.Done()
shutdownCtx, cancel := context.WithTimeout(context.Background(), 25*time.Second)
defer cancel()
srv.Shutdown(shutdownCtx)

Shutdown stops accepting, lets in-flight requests finish, then returns. Keep your own timeout below TimeoutStopSec, or systemd sends SIGKILL in the middle of a graceful stop. Graceful shutdown for a Go HTTP server covers that handshake in detail.

Node can do the same with server.listen({ fd: 3 }) when LISTEN_FDS is set. Other languages have equivalents; the contract is just the two environment variables and descriptor 3.

The measurement

Same setup as our other tests: Debian 12 with systemd 252 as PID 1, Caddy 2.6.2 in front, a Go service that needs about 1.5 seconds to start, 20 requests per second through Caddy the whole time, three runs per scenario. Every fifth request hit a 300 ms endpoint.

A plain systemctl restart of a service behind a socket unit: 161 requests sent, none failed. The same restart without the socket unit lost between 33 and 42 requests of roughly 160. Exactly what you want.

The cost

The requests that arrived during the restart did not fail. They waited, for roughly the restart time, which is around two seconds here: the old process drains first, then the new one starts. Seventeen requests in the plain socket restart took over a second. With short requests and relaxed clients nobody notices.

Then we kept 2-second requests in flight while deploying. The slowest request in that scenario took 5.65 seconds. It was caught behind the stopping process, which waited for its own in-flight work, and then queued behind the starting one. Nothing failed, but a client with a three-second timeout would have given up, and from that client's side it looks like an outage. Blue/green in the same test never made a request wait more than about two seconds, which is the length of the request itself.

The rule we take from this: socket activation converts errors into latency, and the amount of latency is the sum of your drain time and your start time. If that sum is under your clients' timeouts, you are fine.

The other gap: the release itself

Socket activation keeps the door open. It says nothing about whether the new release is any good. Deploy a broken build and restart, and the queue fills with connections that the broken process will never answer.

So you check the release before the restart. We run the new binary as a canary on a spare port with the same environment file, health-check it there, stop it, and only then swap the real service. A crash loop or a 500 from /health is caught before any user is affected.

There is one hole, and we tested it on purpose. A bug that only appears when the process is started by socket activation, for example code that mishandles the inherited descriptor, cannot be caught by a canary that is not socket-activated. In that case the deploy script notices after the restart, restores the previous release and restarts again, which costs a short interruption and a window in which the bad release served traffic. In our run of that scenario the bad release answered 341 or 342 requests before the rollback. The blue/green path has no such window for a release that is broken from the start, because the exact instance that will serve is the one that was checked.

When we would use it

• You cannot run two copies of the service side by side.
• Your requests are short and your clients' timeouts are generous, or clients retry.
• You want the simplest setup that never refuses a connection, and you accept the trade.

If you can run two instances behind a proxy, blue/green is the better default: latency stays flat, the new release is proven in its real role, and a failed switch costs nothing because the old instance never stopped. The switch is laid out in the Caddy blue/green config walkthrough.

Mistakes we see

• Restarting the socket unit. systemctl restart app.socket closes the socket. Restart app.service.
• Enabling the service instead of the socket. The service then starts on boot with its own listener, or fails to bind the port the socket already owns. Enable app.socket.
• Several ListenStream= lines with code that expects one. LISTEN_FDS becomes 2 or more. Decide how your app handles that, or keep one socket per service.
• Testing with tiny requests. A 10 ms request hides the waiting cost entirely. Put your slowest realistic request in the load.
• A default Backlog too small for the traffic. The queue overflows during the restart and clients see resets or timeouts after all.
• Forgetting Type=notify readiness. Without it, systemd considers the service started as soon as it forks, and your deploy script may check health too early. Check against deploy health checks that mean something.

FAQ

Does socket activation work with Caddy or nginx in front?

Yes. Point the proxy at the socket's address, here 127.0.0.1:3000. The proxy sees a listener that never goes away. Connections it opens during a restart are accepted by the kernel and wait in the queue.

Is socket activation the same as systemd starting my service on demand?

It can do that too. If the service is stopped and a connection arrives on the socket, systemd starts the service. For deploys we care about the other property, that the socket outlives the process.

What happens to requests being served when I restart?

With KillSignal=SIGTERM and a Go server that calls Shutdown, the old process finishes its in-flight requests before exiting. New connections wait in the queue meanwhile. That is why a restart with 2-second requests running took longer for the queued ones.

Does it work with HTTP/2 or long-lived connections?

The socket only helps with new connections. Connections already open to the old process end when it stops, so WebSocket and streaming clients still have to reconnect. HTTP/2 connections held by a proxy are also affected until the proxy opens new ones.

Can I get lower latency than 2 seconds with this?

Only by shortening the drain and the start. A faster start-up, a shorter drain timeout and short requests all reduce the wait. The structure, drain then start, stays.

The kit ships both modes. Socket mode includes the canary check, the automatic rollback and a tested sd helper package you can copy into your app, and blue/green is the default unless you choose otherwise. See the Go + systemd Zero-Downtime Deploy Kit for the units, the script and the measurement tables.

deployssystemdgozero-downtime