Starlab

Receiving signed webhooks in Go: verify, dedupe, retry

· 3 min read

A webhook endpoint looks trivial until the sender retries, your handler crashes halfway, or someone replays a captured request. These are the pieces I now put in every receiver, with Go code, and the mistakes each one prevents.

1. Verify the raw body, before anything else

The signature covers the exact bytes the sender sent. Parse JSON first and re-encode it and the bytes change.

body, err := io.ReadAll(http.MaxBytesReader(w, r.Body, 1<<20)) // also caps the size
if err != nil { http.Error(w, "too large", http.StatusRequestEntityTooLarge); return }

Compare in constant time (hmac.Equal), never ==. Reject before touching the database: an unauthenticated request must store nothing.

2. Three header formats you will meet

  • GitHub style: X-Hub-Signature-256: sha256=<hex of HMAC-SHA256(body)>.
  • Stripe style: t=<unix>,v1=<hex>, HMAC over "<t>.<body>", plus a timestamp tolerance so a captured request cannot be replayed next week.
  • Standard Webhooks (Polar uses it): webhook-id, webhook-timestamp, webhook-signature: v1,<base64>, HMAC over "<id>.<timestamp>.<body>", secret is whsec_<base64> and must be base64-decoded first. A header can carry several space-separated signatures (key rotation).

I extracted the verification code, with tests, as a small MIT-licensed module: webhooksig (stdlib only). It is tested against the known-answer vector from the Standard Webhooks reference Go library, with tampered, stale and malformed cases.

3. Answer fast, work later

Senders time out and retry. Verify, store the event, return 202, and process in the background. If you do real work inside the request, a slow handler turns into duplicate deliveries.

4. Dedupe by delivery id, in the database

CREATE TABLE events (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  source TEXT NOT NULL, delivery_id TEXT NOT NULL, payload BLOB NOT NULL,
  status TEXT NOT NULL DEFAULT 'pending', attempts INTEGER NOT NULL DEFAULT 0,
  next_at INTEGER NOT NULL, last_error TEXT NOT NULL DEFAULT '',
  UNIQUE (source, delivery_id)
);
-- INSERT OR IGNORE: the second delivery of the same event inserts nothing

If a provider sends no id, hash the body. A uniqueness constraint beats an in-memory set: it survives restarts.

5. Claim, process, retry with backoff, dead-letter

Claim one ready event atomically (UPDATE ... RETURNING), run your handler, mark it done; on error, schedule a retry with exponential backoff, and park it as dead after N attempts so a poison message cannot loop forever. On startup, return events left processing by a crash to the queue. Your handler must be idempotent, because "at least once" is the best a queue like this can promise.

6. Numbers, honestly

My implementation (SQLite, one writer connection) accepted about 9,000 verified events per second over loopback HTTP on an Apple Silicon laptop and processed about 3,700 per second end to end with a no-op handler. With a handler that sleeps 5 ms (like a network call), 1 worker managed about 160 events per second and 32 workers about 5,500. With a no-op handler extra workers did not help, because everything shares one database connection. I did not measure Linux or cloud disks, large tables, TLS, or long soak runs. Treat these as "this laptop".

What I did not verify

I have not received a real Polar delivery. The Polar event fields in my example come from Polar's public docs and API types, and the older-secret handling (raw: prefix) follows what Polar's docs say. Check one real event before relying on either.

The queue, retries, secret rotation, replay of dead events, metrics and an example Polar handler are packaged as the Go Webhook Receiver Starter ($24): 41 tests with the race detector and the measurements above. The signature module is free either way.