Go in production

log/slog setup for a small Go service on journald

2026-11-09 · 10 min read

Go has had structured logging in the standard library since 1.21, and most small services still use log.Printf with a format string and hope to grep it later. The grep works until the day you need every line belonging to one failed request, in a log that is also full of other requests, at 2 a.m. The fix is cheap: a few dozen lines of setup, once, and every later log call gets better for free.

This post sets up log/slog for a service that runs under systemd on one VPS. It covers choosing JSON or text, keeping the level changeable without a rebuild, attaching a request id to every line without passing it by hand, keeping secrets out, and then reading it all back with journalctl. The code compiles on Go 1.22 and later.

Who needs this

A small API with a handful of routes. A user reports "it failed around 14:10". With a request id in the response header and on every log line, you search for that id and see the whole request: the route, the database call, the error. Without it you are matching timestamps by eye.

A service with background workers. The worker and the HTTP handlers write to the same stream. Structured fields (job_id, attempt, worker) let you filter one job's history out of thousands of lines, which a free-text message cannot do reliably.

A service that handles tokens and credentials. Webhook secrets, API tokens and session cookies pass through your code. One careless log.Printf("%+v", cfg) puts them in the journal, and the journal gets copied into bug reports. A redaction layer and a safe Config type make that mistake harmless.

A solo operator watching one box. No log shipper, no Loki. Journald is the log store, and journalctl plus jq is the query language. The log format needs to be friendly to exactly that.

The setup, in one place

package logx

import (
	"context"
	"io"
	"log/slog"
	"strings"
)

type ctxKey struct{}

// WithRequestID returns a context that carries the request id.
func WithRequestID(ctx context.Context, id string) context.Context {
	return context.WithValue(ctx, ctxKey{}, id)
}

// RequestID returns the id stored by WithRequestID, or "".
func RequestID(ctx context.Context) string {
	id, _ := ctx.Value(ctxKey{}).(string)
	return id
}

// ctxHandler decorates another handler: it adds request_id from the context.
type ctxHandler struct{ slog.Handler }

func (h ctxHandler) Handle(ctx context.Context, r slog.Record) error {
	if id := RequestID(ctx); id != "" {
		r.AddAttrs(slog.String("request_id", id))
	}
	return h.Handler.Handle(ctx, r)
}

func (h ctxHandler) WithAttrs(a []slog.Attr) slog.Handler { return ctxHandler{h.Handler.WithAttrs(a)} }
func (h ctxHandler) WithGroup(n string) slog.Handler      { return ctxHandler{h.Handler.WithGroup(n)} }

var secretKeys = []string{"token", "secret", "password", "authorization", "cookie", "api_key"}

// redact replaces the value of any attribute whose key looks like a secret.
// It is a safety net, not a substitute for not logging secrets in the first place.
func redact(groups []string, a slog.Attr) slog.Attr {
	k := strings.ToLower(a.Key)
	for _, s := range secretKeys {
		if strings.Contains(k, s) {
			return slog.String(a.Key, "[redacted]")
		}
	}
	return a
}

// Level can be changed while the process runs.
var Level slog.LevelVar

// New returns a logger writing to w. format is "json" or "text".
func New(w io.Writer, format string) *slog.Logger {
	opts := &slog.HandlerOptions{Level: &Level, ReplaceAttr: redact}
	var h slog.Handler
	if format == "text" {
		h = slog.NewTextHandler(w, opts)
	} else {
		h = slog.NewJSONHandler(w, opts)
	}
	return slog.New(ctxHandler{h})
}

That is the whole package. The shape follows the logging chapter of our Go handbook, with one change here: the level is a slog.LevelVar so it can move at runtime. In main:

logger := logx.New(os.Stdout, os.Getenv("LOG_FORMAT")) // "" means json
logx.Level.Set(slog.LevelInfo)
slog.SetDefault(logger)

slog.SetDefault also redirects the old log package, so a dependency that still calls log.Printf ends up in the same stream, at info level, instead of in a separate unstructured corner. Everything below explains why each piece is there.

JSON or text

Use JSON in production and text on your laptop. The decision is not about taste:

• JSON gives every field a name. jq can select on .level, .request_id or .duration_ms, and any future log shipper reads it without a parser. A single line is still readable when you know what to look for.
• Text (key=value pairs) is far easier to read in a terminal while you develop. It is also fine in production if you will only ever use grep.

Pick the format from an environment variable so the same binary does both. The default should be JSON, because the failure mode of a wrong default is a production service that logs in a format you cannot query. With the code above and the same two log calls, JSON looks like this:

{"time":"2026-11-09T17:12:46.659904+07:00","level":"INFO","msg":"loading","id":"7","request_id":"48a6e95946170344"}

and the text handler prints:

time=2026-11-09T17:12:47.273+07:00 level=INFO msg=loading id=7 request_id=9ca6d7bdf33e28eb

The reference service reads REFSVC_LOG_FORMAT=text for the second form and uses JSON otherwise.

Request ids without threading them through every function

Passing a request id as a parameter to every function is how people end up deleting the request id. The context already travels through every call that matters, so put the id there once, in a middleware, and let the handler wrapper above add it to each record.

func withRequest(log *slog.Logger, next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		id := r.Header.Get("X-Request-Id")
		if !validID(id) {
			var b [8]byte
			_, _ = rand.Read(b[:])
			id = hex.EncodeToString(b[:])
		}
		w.Header().Set("X-Request-Id", id)
		ctx := logx.WithRequestID(r.Context(), id)
		r = r.WithContext(ctx)

		sw := &statusWriter{ResponseWriter: w}
		start := time.Now()
		defer func() {
			if sw.status == 0 {
				sw.status = http.StatusOK
			}
			log.LogAttrs(ctx, slog.LevelInfo, "request",
				slog.String("method", r.Method),
				slog.Int("status", sw.status),
				slog.Float64("duration_ms", float64(time.Since(start).Microseconds())/1000))
		}()
		next.ServeHTTP(sw, r)
	})
}

// validID accepts only short, boring ids so a client cannot inject log noise.
func validID(s string) bool {
	if s == "" || len(s) > 64 {
		return false
	}
	for _, c := range s {
		ok := c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z' || c >= '0' && c <= '9' || c == '-' || c == '_'
		if !ok {
			return false
		}
	}
	return true
}

type statusWriter struct {
	http.ResponseWriter
	status int
}

func (w *statusWriter) WriteHeader(code int) { w.status = code; w.ResponseWriter.WriteHeader(code) }
func (w *statusWriter) Unwrap() http.ResponseWriter { return w.ResponseWriter }

The imports are crypto/rand, encoding/hex, log/slog, net/http and time, plus your logx package. A few decisions in there deserve explaining.

An inbound X-Request-Id is accepted only if it is short and made of letters, digits, - and _. Otherwise a client can put newlines, quotes or a 10 KB blob into your logs. If the proxy in front of you generates ids, this lets them flow through. If not, you make one: eight random bytes, hex encoded, is plenty to search for.

The id goes back out in the response header. A user who reports a failure can paste it, and that is the whole value of the feature.

The access line logs method, status and duration, and deliberately not the query string, headers or body. A query string is where tokens end up (?token=...), so leaving it out removes a whole class of leak. If you are on Go 1.23 or later, r.Pattern holds the matched route such as GET /x/{id}; log that instead of the raw path, so ids inside paths stay out of the log and the same endpoint always has the same label. On 1.22 the field is not available, so log the method and let handlers add their own context.

Inside a handler, the rule is one habit: use the Context variants.

logger.InfoContext(r.Context(), "loading job", "job_id", id)
logger.ErrorContext(r.Context(), "query failed", "err", err)

A call to plain logger.Info has no context, so it carries no request id. That is the one thing to teach every contributor, and a code review is where it gets caught.

Levels that mean something

Four levels, each with a job:

Two rules keep the signal useful. A client's mistake (bad JSON, 404) is not an ERROR, because that turns your error count into a measure of other people's typos. And log an error once, where you handle it, not at every layer it passes through. The usual pattern is to wrap with fmt.Errorf("load job %d: %w", id, err) on the way up and log at the top.

Run at INFO. Changing the level should not need a rebuild or a code change. With the slog.LevelVar in the setup you have two easy options. Read it from an environment variable at startup (LOG_LEVEL=debug, applied with Level.UnmarshalText), or expose a loopback-only admin endpoint that calls Level.Set. For a single-VPS service the first is enough: systemctl edit myapp, add Environment=LOG_LEVEL=debug, restart, investigate, remove the line. If a restart would destroy the state you are trying to observe, that is when the admin endpoint pays for itself.

Keep secrets out, in two layers

Rule of thumb: the redaction filter is a net for mistakes, not a reason to log secrets.

The first layer is not logging them: no headers, no bodies, no query strings in the access line. The second is a safety net for the mistake that happens anyway.

redact above runs as a ReplaceAttr function. It is called for every attribute, including those inside groups, and replaces the value of any key that contains token, secret, password, authorization, cookie or api_key. An attribute named Authorization or api_token comes out as [redacted].

A key-name filter cannot see a secret hiding in a message string or under an innocent name, so it is only a net. The stronger measure is to make the dangerous type safe by itself. If your config struct holds the API token, implement slog.LogValuer on it:

type Config struct {
	Addr     string
	APIToken string
}

// LogValue means logging a Config can never leak the token.
func (c Config) LogValue() slog.Value {
	return slog.GroupValue(
		slog.String("addr", c.Addr),
		slog.String("api_token", "[redacted]"),
	)
}

Now logger.Info("starting", "config", cfg) prints "config":{"addr":":8080","api_token":"[redacted]"} and there is no code path where the real token is printed. That is the kind of fix that is done once and stays done. We cover the loading and validation of such a config in the production handbook.

The standard library's own logs

net/http logs some things itself (accept errors, panics in handlers, TLS handshake failures) through Server.ErrorLog, which defaults to the old logger writing plain text to stderr. Route it through slog so those lines are structured too:

srv := &http.Server{
	Handler:  handler,
	ErrorLog: slog.NewLogLogger(logger.Handler(), slog.LevelWarn),
}

slog.NewLogLogger is in the standard library, and the second argument is the level those lines get. Combined with explicit server timeouts, this is where you will see slow-client noise show up, at warning level, next to your own lines.

Reading it back with journalctl

Under systemd, standard output goes to journald, so there is no log file to manage and no rotation code to write. Each JSON line becomes the MESSAGE of a journal entry. A few queries cover most days:

# follow the service
journalctl -u myapp -f -o cat

# everything for one request id
journalctl -u myapp --since "1 hour ago" -o cat \
  | jq -cR 'fromjson? | select(.request_id == "48a6e95946170344")'

# only errors, last day
journalctl -u myapp --since yesterday -o cat \
  | jq -cR 'fromjson? | select(.level == "ERROR")'

-o cat prints just the message, which is your JSON. jq -R reads raw lines and fromjson? skips any line that is not JSON, such as a panic trace the runtime wrote to stderr, instead of aborting.

Be aware of one limit. Journald assigns a priority to each entry, and for ordinary stdout lines it is info unless the line starts with a syslog-style prefix like <3>. That means journalctl -p err will not find your ERROR lines, because slog writes none of those prefixes. Filter on the JSON level field as above. If you want native priorities, a small io.Writer wrapper can prepend <3> to lines containing "level":"ERROR", but then your output is no longer pure JSON for tools downstream, so decide which matters more.

Volume is the other thing journald has opinions about. It rate-limits a service that logs too fast (the defaults in journald.conf are 10,000 messages per 30 seconds per service) and it has a size cap on the whole journal. One access-log line per request is fine for a small service and expensive for a busy one. If the disk fills, see journald and logrotate when the disk is full, and for a lightweight view across the machine, monitoring one VPS without a stack.

Mistakes we see

Two loggers in one process. The log package for old code and slog for new, each with its own format. slog.SetDefault plus passing the logger where you need it removes the split.

fmt.Sprintf inside the message. logger.Info(fmt.Sprintf("job %d failed", id)) puts the id in free text. Use logger.Info("job failed", "job_id", id) so the field is queryable.

Odd numbers of key-value arguments. logger.Info("x", "key") produces a !BADKEY attribute. go vet has a slog check that catches these; run it in CI.

Logging whole structs or requests. logger.Info("got", "req", r) can dump headers, cookies and bodies. Log the fields you mean.

Per-request debug lines left on. A hot path logging at debug is free while the level is info, but a DEBUG run in production on a busy service fills the journal in an hour. Turn it on, reproduce, turn it off.

Using the global logger inside handlers with no context. The line has no request_id. Use the Context methods.

FAQ

Should I use slog or zap or zerolog on a small service?

slog is enough for a service this size and has no dependency. The third-party libraries have their own strengths, mostly around allocation counts at very high log volume, but for a one-VPS service the cost of a dependency and a second API style outweighs it. slog's Handler interface also lets you swap the backend later without touching call sites.

How do I add a field to every log line, like the version?

Create the logger with With: logger := logx.New(os.Stdout, format).With("version", version). Every record from that logger, and any logger derived from it, carries the field.

Can I change the log level without restarting?

Yes, with a slog.LevelVar. Hold the pointer in the handler options, then call Set from wherever you choose: a signal handler, a loopback admin route, or a config reload. Only expose it on an address that is not public.

Is the time field needed if journald already timestamps entries?

Journald's timestamp is when it received the line, which is almost always close enough. The slog time is when the event happened in your process. Keep it. It costs a few bytes and it survives if you ever copy the logs elsewhere. If you want to drop it, return an empty slog.Attr{} from ReplaceAttr when a.Key == slog.TimeKey and len(groups) == 0.

How do I find out what slog costs in allocations?

Use go test -bench with -benchmem on a handler writing to io.Discard. Calls whose level is disabled are very cheap, and LogAttrs avoids some boxing compared with the any-based key-value form on hot paths. For a typical small service the log call is not the bottleneck.

The Go on One VPS handbook has this as a chapter with the tested logger package, the request-id middleware, the redaction tests and the check that the API token never reaches the journal under systemd: Go on One VPS: Production Handbook.

gojournald