Postgres Replication Slots Hold WAL Until Disk Fills
A replication slot is a promise: Postgres will keep the WAL a consumer still needs. When the consumer disappears, the promise outlives its purpose, and the WAL directory grows until the disk fills. This piece covers how slots pin segments, how the failure unfolds, and the settings and habits that keep retention bounded.
The Slot That Ate The Disk
Replication slots were added to Postgres to solve a real problem: a standby or logical consumer could fall behind and the primary would recycle WAL it had not yet received. The slot tracks the consumer's position and prevents removal of segments beyond it. That guarantee is the point, and it is also the hazard.
The slot does not know whether the consumer is alive, paused, or gone. It knows only the last confirmed position. An inactive slot holds that position indefinitely by default. There is no built-in timeout, and no warning that escalates with age. A slot created for a migration that finished months ago will still pin every segment written since.
The failure is quiet until it is not. Disk usage climbs, checkpoints begin to fail, and the write path hits a PANIC. A single forgotten slot can halt a primary that has run cleanly for years. The fix is usually one pg_drop_replication_slot call, but reaching it requires a working connection and enough free space to operate.
The pattern repeats often enough that it has a recognizable shape. A team stands up a logical replica for an analytics pipeline, points it at the primary, and the pipeline gets deprecated six months later. The replica host is decommissioned. The slot remains, because the slot lives on the primary and nobody dropped it. WAL keeps accumulating at whatever rate the workload produces, and the volume fills weeks or months later, long after the person who created the slot has moved to another project. The delay between cause and symptom is what makes this failure so easy to miss.
How Slots Pin WAL Segments
Each slot records two positions: restart_lsn, the oldest WAL still needed, and confirmed_flush_lsn, the last position the consumer acknowledged. For logical slots, confirmed_flush_lsn advances only when the consumer confirms receipt. Physical slots advance through feedback from the standby.
The WAL recycler computes a retention bound from the minimum restart_lsn across all slots. Segments newer than that bound stay in pg_wal, even after a checkpoint would otherwise recycle them. The directory grows in 16 MB segments by default, so a slot lagging by a few gigabytes is visible as a few hundred files.
Postgres does expose a warning. When retention exceeds max_slot_wal_keep_size, the slot's wal_status moves to reserved, then eventually lost, and the slot is invalidated. The default for that setting is -1, meaning unlimited. That default is the reason the failure mode exists at scale.
It helps to separate the two slot kinds here, because they fail differently. A physical slot tracks a standby's replay position, and if the standby is reachable, feedback keeps the position moving. A logical slot tracks a decoding consumer, and its confirmed_flush_lsn only moves when that consumer explicitly confirms. A logical consumer that connects, reads a little, and then stalls without confirming leaves the slot pinned at the last confirmation, not at the last byte it read. The distinction matters when you are reading the view and trying to decide whether a slot is genuinely stuck or merely a few seconds behind.
There is also a subtler retention source: wal_keep_size and the archive settings can keep segments around independently of slots. When disk fills, confirm which mechanism is actually holding the segments before you start dropping things. The pg_replication_slots view tells you the slot side of the story; the archive configuration tells you the rest.
When The Failure Arrives
The sequence is predictable. WAL accumulates until the filesystem reports no free space. The next checkpoint cannot write, and the checkpointer process exits. Writes then fail with a PANIC, and the postmaster begins a crash recovery cycle that cannot complete because it needs to write WAL.
Reads may survive briefly on an already-open connection, but new connections typically fail during recovery. The server is effectively down, and the usual administrative tools need the same disk the server cannot write to. Teams often free space by removing old base backups or logs before they can drop the slot.
Recovery requires dropping the offending slot or moving pg_wal to a larger volume. Dropping the slot immediately recycles WAL up to the next consumer's position, which can free space in seconds. If the slot was a logical consumer that still needs data, that data is gone, and the consumer must be rebuilt from a fresh copy.
The rebuild is the part teams underestimate. A logical consumer that has to resync from scratch reads a full copy of the tables it tracks, which on a large database can take hours and load the primary with the read traffic. If the consumer was feeding a downstream system, that system is stale for the duration. Budget for the resync window when you decide whether to cap retention or let it run unbounded.
Monitoring The Right Signals
pg_replication_slots is the primary view. The active column shows whether a consumer is currently connected. A slot with active = false for more than a few hours is a candidate for investigation. The restart_lsn and confirmed_flush_lsn columns show the lag in bytes when compared to the current WAL position.
Directory size matters more than any single metric. Track the total size of pg_wal against free space on the volume, and alert when the directory exceeds a fixed threshold or when free space drops below a margin. The pg_stat_replication view covers connected standbys and their replay lag, but it says nothing about disconnected slots.
The gap between pg_wal growth and disk capacity is the real signal. A slot lagging by tens of gigabytes on a volume with hundreds free is survivable; the same lag on a smaller volume is an outage waiting. Set thresholds from the volume size, not from a generic number copied between clusters.
Two derived numbers are worth computing on a schedule. The first is the retention distance: the difference between the current WAL insert position and the oldest restart_lsn, converted to bytes. The second is the time to fill: that distance divided by the recent WAL generation rate, which you can estimate from the growth of pg_wal over an hour or a day. Time to fill is the number that tells you whether you have a week or an afternoon. A slot that has been inactive for a month on a busy primary may have a time-to-fill measured in hours, and that is the case worth paging on.
Watch the wal_status column too. A slot that has moved to reserved is already inside the retention bound and heading for invalidation; that transition is a leading indicator, not a routine state.
Coping Patterns That Work
max_slot_wal_keep_size is the direct control. Setting it to a disk-aware bound, such as a fraction of the volume, lets Postgres invalidate a slot that exceeds retention rather than fill the disk. The trade-off is real: the slot becomes unusable and the consumer must resync. For a logical replica that is already far behind, that may be the cheaper outcome.
Temporary slots suit one-off logical copies. Create the slot, run the copy, drop the slot in the same script. A slot that exists only for the duration of a job cannot be forgotten. For long-lived consumers, pair every slot with an owner and an expiry date recorded somewhere outside the database.
pg_waldump and the pg_replication_slots view together identify what a slot is for. If the slot name does not map to a known consumer, treat it as abandoned. Before decommissioning a replica, drop its slot first; the reverse order leaves an orphan. The same discipline applies to client libraries that manage connections, as this site has noted in Postgres wire protocol changes.
The choice between capping retention and leaving it unlimited is a choice about which failure you prefer. With a cap, you trade a slot invalidation and a resync for a bounded disk footprint; you accept that a long-stalled consumer loses its position and must be rebuilt. Without a cap, you keep every consumer's position intact and accept that a forgotten slot can take down the primary. Neither is free. The cap converts an unbounded outage into a bounded, recoverable event, and for most production clusters that is the right trade.
Operational Habits To Adopt
- Audit pg_replication_slots weekly and flag any slot with active = false for more than a day.
- Set max_slot_wal_keep_size to a value derived from the volume size, not copied from another cluster.
- Alert when pg_wal exceeds a fixed size threshold or when free space on the volume drops below a defined margin.
- Record an owner and an expiry date for every slot at creation, and review the list monthly.
- Test dropping a slot and recovering from the resulting WAL recycle in a staging cluster before you need to do it under pressure.
These habits are unglamorous, and they cost a few minutes a week. The alternative is a primary that stops accepting writes at an inconvenient hour because a slot nobody remembers is still holding a promise. A related piece on certificate pinning makes a similar point about silent failures that surface only when the margin is gone.