Postgres and data

How much data a disk failure costs with WAL archiving

2026-11-09 · 10 min read

"We have point-in-time recovery" usually means "we can restore to any moment that has already reached the archive". Those are different sentences, and the distance between them is how much you lose when the disk goes. On a default PostgreSQL install with archiving turned on, that distance can be everything since your last base backup.

We measured it, on purpose, by killing a server and recovering from the archive alone. This post gives the numbers, explains where the loss comes from, and shows how to pick a setting for your own tolerance.

The test

The workload was a PostgreSQL 15 server taking 20 single-row inserts per second, with WAL archiving to a separate directory. After some seconds of work we ran kill -9 on the server, threw away its data directory, and restored from the base backup plus the archive only, as if the disk had died. Then we counted how many of the acknowledged inserts were missing. An insert counts as acknowledged when the client got its commit back. If the client was told "done", the row should be there.

The runs used the scripts from the Postgres PITR Kit, in a Docker container on a laptop-class ARM64 machine. Treat the numbers as orders of magnitude, not as a benchmark. Your disk, your WAL volume and your archive target will move them.

Why anything is missing at all

PostgreSQL writes every change to a write-ahead log made of 16 MiB segment files in pg_wal/. The server hands a segment to archive_command only when the segment is finished. A segment is finished when it fills up, or when something forces a switch to the next one: a manual pg_switch_wal(), a base backup starting or stopping, or the archive_timeout setting.

On a busy server a segment fills in seconds. On a quiet one it can take hours. Everything committed since the last switch lives only in the current segment, in pg_wal/ on the disk that just died. The archive has no idea it exists.

Two separate things are going on here, and people mix them up:

• Durability of the commit. When PostgreSQL acknowledges a commit, the record is flushed to the local WAL. A crash and restart on the same disk loses nothing.
• Availability of that record elsewhere. The record is copied to the archive only when its segment ships. Until then, a lost disk takes it.

archive_timeout is the lever for the second one. Set it to N seconds and the server switches to a new segment when N seconds have passed since the last switch and there was some activity. It does not switch an idle server, so a quiet night does not produce a pile of empty files.

The numbers

Rule of thumb: the loss is bounded by archive_timeout plus shipping time, and with the default of 0 it is not bounded at all.

Read the last row twice. With the default setting, on a server that is not very busy, a lost disk loses everything since the last base backup, even though "WAL archiving is on". The recovery itself works and the database starts. It just comes back at the moment the base backup was taken, because nothing newer was ever shipped.

The 10 second rows show the bound we were after: the newest lost commit was between 1.9 and 6.1 seconds newer than the newest recovered one, which is under archive_timeout plus the time to ship a file. The 60 second row lost up to 36 seconds of work. In both cases the loss is bounded by the timeout, and in neither case is it zero.

What a low archive_timeout costs

Every forced switch archives a full 16 MiB file, however little is in it. Without compression, archive_timeout = 60 on a server that sees activity every minute is up to 1,440 files a day, about 22.5 GiB, whatever your real write volume. That is the cost people hear about and then avoid by leaving the setting at 0.

Compression changes the picture, because a nearly empty segment is nearly all zeros. In our runs a gzip-compressed, almost empty segment came to about 16 KB. Over a whole RPO run, 3 files for the 60 second case weighed 180 KiB in total, and 4 to 6 files for the 10 second cases weighed 180 to 1,488 KiB (the first file is bigger), against 16 MiB per file uncompressed.

For a full segment, the compression cost is time, not space. We archived one real, full 16 MiB segment from a pgbench run, five calls each:

That pgbench run produced about 2.1 MiB of WAL per second, so a segment filled roughly every 7.6 seconds. Gzip at 0.35 s per 7.6 s is about 5 percent of one core. That figure is arithmetic from the two measurements above, not a separate benchmark, but it tells you the order of magnitude: you can compress WAL on a small VPS without noticing.

WAL volume, so you can estimate your own

The amount of WAL you produce decides how often segments fill by themselves. These are differences of pg_current_wal_lsn() before and after each workload:

To get yours, run this twice, a day apart, and subtract:

select pg_current_wal_lsn();
select pg_size_pretty(pg_wal_lsn_diff('0/5000060', '0/3000000'));

If your server writes less than 16 MiB between the moments you care about, archive_timeout is what decides your loss, not the segment size.

Does archiving slow the server down?

We looked for it with pgbench: scale 10 (about 157 MB), 4 clients, 30 seconds per run, five rounds, with the three configurations interleaved in every round.

The medians differ by less than 1 percent, inside the roughly 2 percent spread between rounds. No overhead was detectable in that setup, where the working set fits in memory. Do not read more into it than that. On a shared machine the same test is noisy, and you should measure on your own hardware and workload.

Who needs this

A small SaaS on one VPS. Orders, accounts and invoices live in one Postgres database on one box. Nightly pg_dump means a failure at 17:00 costs 15 hours of customer data. With archiving and archive_timeout = 60 the same failure costs about a minute. For this project the interesting question is the price: a few hundred megabytes of compressed WAL a day is nothing, and the base backup is a nightly cron job.

A low-traffic internal tool. A dozen people use it, a few hundred writes a day. This is the project in the last row of the table. Segments fill in days, so archiving "on" with default settings is close to no protection at all. Setting archive_timeout to something between 60 and 300 seconds costs almost nothing here and moves you from "the last base backup" to "a few minutes ago".

A write-heavy service ingesting events. Segments fill in seconds on their own, so archive_timeout rarely fires and your loss is dominated by how fast the archive can keep up. The risk here is the other direction: a slow or full archive makes WAL pile up in pg_wal/ and fills the data disk. Watch the lag, as described in monitoring backups so silence is not success.

A team that must answer "how much can we lose?" in writing. Customers, auditors and your own manager want a number. "Up to archive_timeout plus shipping time, measured on this date by a drill" is a defensible answer. "We have backups" is not.

Choosing the setting

Start from the loss you can stand, then check the cost.

1. Decide your recovery point objective (RPO) in plain units. Five minutes? One minute? Write it down.
2. Set archive_timeout to about half of it. The shipping time and the gap before the next switch add to the timeout, and you want slack.
3. Turn on gzip for archived segments so the idle-server cost stays small.
4. Run a loss drill like the one above on a scratch copy, so the number in your runbook is yours and not ours.
5. Alert on the age of the newest archived segment, so you find out the day shipping stops and not the day the disk dies.

A practical floor is about 10 seconds. Below that you are asking a general-purpose archive pipeline to do the job of replication, and you should use replication.

If you want zero loss

Then archiving is the wrong tool. These are the usual options, in order of effort:

Streaming options are a different operational commitment: a second server, a network path that has to stay up, and a decision about what happens to writes when the standby disappears. For most one-box projects, a well-chosen archive_timeout is the right trade, and the honest statement is "we can lose up to N seconds".

Mistakes we see

Treating "archiving is on" as the finish line. archive_mode = on with the default archive_timeout = 0 proves a file got copied once. It says nothing about how recent the newest archived record is. Query pg_stat_archiver and look at last_archived_time.

Archiving to the same disk. If archive_command writes to a directory on the data disk, the loss test in this post stops being a test: the archive dies with the server. Use another disk, a mounted volume, or another host.

Measuring RPO from the schedule instead of from a drill. A cron entry that says "every minute" tells you what you asked for. A drill tells you what arrived.

Setting archive_timeout without compression on a tiny archive disk. A 10 second timeout without gzip can write several gigabytes a day of mostly empty files. Compress, or size the archive for it.

Forgetting that the timer only counts when there is activity. An idle server does not switch segments, and that is fine, but it means a burst of writes right before the failure can sit in the current segment too.

FAQ

What is the default archive_timeout in PostgreSQL?

It is 0, which disables the timer. With archiving on, a WAL segment is shipped only when it fills (16 MiB by default) or when something forces a switch. On a quiet server that can be hours or days.

Does archive_timeout guarantee I lose at most that many seconds?

No. It is a bound on how long a record waits before its segment is closed, plus the time archive_command takes to ship it, plus any backlog if the archive was slow or failing. In our 10 second runs the newest lost commit was 1.9 to 6.1 seconds newer than the newest recovered one. Treat it as a target you verify with a drill.

Will a short archive_timeout fill my disk?

It adds one full-size segment per switch, so uncompressed it can. At 60 seconds that is up to about 22.5 GiB a day. Compressed, a mostly empty segment is a few kilobytes, so the archive stays small. The danger to watch is a failing archive, not a short timeout: unarchived segments stay in pg_wal/ until they ship.

Does WAL archiving slow down my database?

In our pgbench comparison the medians for archiving off, gzip and plain copy were within 1 percent of each other, below the spread between rounds. Your write volume and disk decide your own answer, so measure on your hardware.

How do I get to zero data loss?

Use synchronous replication to a standby, or stream WAL with pg_receivewal. Archiving ships finished files, so it always has a gap. See pg_dump vs physical backup vs PITR for where each approach fits, and WAL archiving from scratch for the setup.

The Postgres PITR Kit sets archiving up with a sane archive_timeout, monitors the age of the newest archived segment, and runs the whole destroy-and-restore drill so you can see your own numbers: Postgres PITR Kit.

postgreswalbackuppitrrecovery