Pilotbase on a PostgreSQL 18 cluster restored to just before a bad DELETE: 6,003,069 orders, 5,909,287 of them dated before 1 October, last payment at 00:41:43.226883 UTC

Someone meant to purge orders older than three years and typed 2026-10-01 instead of 2023-10-01. The DELETE took 6.3 seconds and removed 5,909,287 rows. Last night's pg_dump would bring them back, along with the loss of every order and payment since midnight. What you want is the database as it was one moment before that DELETE committed, and that's what point-in-time recovery (PITR) gives you.

We ran the whole thing on PostgreSQL 18.6 with pgBackRest 2.60: a 7.1 GB database taking about 100 transactions a second, a full backup, the bad DELETE, finding its exact commit in the WAL with pg_waldump, and a restore that stopped one transaction short of it. Every number below comes from that run, on an Apple silicon Mac mini (10 cores, 16 GB, internal SSD) with the backup repository on the same disk.

Why a dump can't do this

pg_dump takes a logical snapshot: the database as it was when the dump started, and nothing after. Restoring one is worth testing (how we test a pg_dump restore), but it can only take you back to that moment. PITR works on two other pieces:

text
1. a physical base backup: a copy of the data directory's files, taken while the server runs
2. the WAL archive: every write-ahead log segment (16 MB each) since that backup, copied off the server

restore = put the base backup back, then replay WAL up to the moment you choose

The WAL already records every change the server makes, in commit order, with a timestamp on every commit. Archiving it just means keeping the segments somewhere safe instead of letting the server recycle them. That's the part that has to be set up before the accident: without an archive there is nothing to replay.

The setup: archive_mode, archive_command and a pgBackRest stanza

Three server settings. wal_level is already replica by default; archive_mode needs a restart, archive_command only a reload:

ini
# postgresql.conf
wal_level = replica              # the default
archive_mode = on                # needs a restart
archive_command = 'pgbackrest --stanza=main archive-push %p'
archive_timeout = 60             # close a segment at least once a minute on a quiet server

archive_timeout matters on a quiet database: a segment is archived only when it fills or is switched, so a server writing a few kilobytes a minute could sit on an unarchived hour. With 60 s, a lost server costs you at most about a minute of commits. pgBackRest compresses each forced segment, so the mostly empty ones cost almost nothing.

pgBackRest's own config names the repository and the cluster. This is the one we ran (a local repository, zstd, four processes):

ini
# /etc/pgbackrest/pgbackrest.conf
[global]
repo1-path=/var/lib/pgbackrest
repo1-retention-full=2
compress-type=zst
compress-level=3
process-max=4
start-fast=y

[main]
pg1-path=/var/lib/postgresql/18/main
pg1-port=5432
text
$ pgbackrest --stanza=main stanza-create
$ pgbackrest --stanza=main check          # forces a WAL switch and waits for it to reach the archive
$ pgbackrest --stanza=main --type=full backup
$ pgbackrest --stanza=main info

check is worth running every time you change anything: it fails loudly if archive_command can't write to the repository, which is otherwise only visible as archive_status/*.ready files piling up in pg_wal until the disk fills.

The test database and the backup

text
database                         7.1 GB  (pgbench scale 400: 40 M accounts, plus 6 M orders with text notes)
workload                         ~100 tx/s: 75% pgbench TPC-B, 25% INSERT INTO orders

pgBackRest full backup           12.7 s     7.1 GB -> 850.5 MB in the repository
pg_basebackup -Ft, server zstd:3 12.9 s     7.1 GB -> 894 MB base.tar.zst + 16 MB pg_wal.tar
WAL while the workload ran       one 16 MB segment every ~11 s, 5.45 MB each after zstd (32.5%)

Take the backup sizes with a pinch of salt. pgbench's accounts table is mostly blank padding, so its 5.9 GB compressed to 273 MB on its own; the 1.3 GB orders table, full of random text, was most of the rest. Your data will compress less. The WAL figure is the more useful one: about 1.5 MB/s of WAL for 100 small transactions a second, a third of it after compression.

The accident

Two minutes into the workload, the purge ran:

text
DELETE FROM orders WHERE created_at < '2026-10-01';   -- meant '2023-10-01'

started      00:41:37.08 UTC
took         6.28 s
rows         5,909,287 of 6,002,917
WAL          ~81 segments, about 1.3 GB, for one statement

Nobody noticed for three minutes. In that time the application kept working: 4,309 new orders and 12,941 payment transactions went into the damaged database. Hold on to those two numbers; they come back at the end.

Finding the exact moment with pg_waldump

A time from an application log or a Slack message ("around 00:41") isn't good enough. The target has to sit between the last transaction you want to keep and the DELETE's commit, and on a busy server those can be milliseconds apart. The WAL has the exact answer.

First narrow it down to a segment. Each archived segment's commits carry timestamps, so a loop that prints the first and last commit time per segment finds the right file in seconds. pgBackRest's archive-get fetches and decompresses one segment; pg_waldump reads it. Set TZ, because pg_waldump prints commit times in the machine's local time zone:

text
export TZ=UTC
mkdir -p /tmp/wal && cd /tmp/wal
pgbackrest --stanza=main archive-get 000000010000000400000028 /tmp/wal/000000010000000400000028
pg_waldump --rmgr=Transaction 000000010000000400000028 | head -3

In our archive, segments with a commit every few milliseconds went quiet around 00:41:37: about a thousand commits per segment dropped to a few dozen or none for nearly 1.3 GB of WAL, which is what one long statement crowding out everything else looks like. The commit that ended it:

text
rmgr: Transaction ... tx:      13232, lsn: 4/25D561E8, ... desc: COMMIT 2026-10-26 00:41:43.228894 UTC
rmgr: Transaction ... tx:      12586, lsn: 4/28699A28, ... desc: COMMIT 2026-10-26 00:41:43.351658 UTC   <- the DELETE
rmgr: Transaction ... tx:      13233, lsn: 4/2869D998, ... desc: COMMIT 2026-10-26 00:41:43.400560 UTC

To be sure 12586 is the DELETE and not some other big transaction, look at its records. The table's file node comes from SELECT pg_relation_filenode('orders') and the database OID from pg_database; pg_waldump's --relation filter takes tablespace/database/filenode:

text
$ pg_waldump -p /tmp/wal -s 4/22000000 -e 4/28699A60 --relation=1663/5/16414 | grep -m1 'tx:      12586'
rmgr: Heap  len (rec/tot): 54/54, tx: 12586, lsn: 4/22000218, ... desc: DELETE xmax: 12586, off: 2,
      infobits: [KEYS_UPDATED], flags: 0x00, blkref #0: rel 1663/5/16414 blk 117047

Heap DELETE records on orders, from transaction 12586, committed at 00:41:43.351658 at LSN 4/28699A28. Notice the transaction IDs around it: 13232 committed before 12586 did. Transaction IDs are handed out when a transaction starts writing, and the DELETE started six seconds before it finished, so hundreds of transactions that started after it (IDs up to 13232) committed while it ran. Recovery follows commit order, not ID order, which is why all of them are kept below.

Three ways to name the stopping point, and the one that bites

text
target                                        stops                                   keeps the DELETE?
recovery_target_time = '...43.351658+00'      after commits at that time (inclusive on)  YES: it replays it
  + recovery_target_inclusive = off           just before the DELETE's commit            no
recovery_target_xid = '12586'                 after 12586 (inclusive on)                 YES
  + recovery_target_inclusive = off           just before 12586 commits                  no
recovery_target_lsn = '4/28699A28'            + inclusive = off                          no
recovery_target_time = '00:41:43+00'          before 00:41:43.000                        no, but loses 37 commits

recovery_target_inclusive defaults to on, so the exact commit time or XID of the DELETE, used as is, replays the DELETE too. Either set inclusive to off, or pick a time just before the commit. Rounding down to the second looks safe and is, but it isn't free: our archive had 37 commits between 00:41:43.000 and the last good one at 43.228894, and a target of 00:41:43 throws them away. Always write the time zone as a numeric offset; the PostgreSQL 18 docs say abbreviations like IST or EST are refused.

The restore: 28 seconds of files, 20 seconds of replay

The quickest route puts the cluster back in place. Stop PostgreSQL, then let pgBackRest restore with --delta, which checksums the files already in the data directory and only rewrites those that differ:

text
$ pg_ctlcluster 18 main stop        # or systemctl stop postgresql@18-main
$ sudo -u postgres pgbackrest --stanza=main --delta \
      --type=time "--target=2026-10-26 00:41:43.351658+00" --target-exclusive \
      --target-action=promote restore
INFO: repo1: restore backup set 20261026-060911F, recovery will start at 2026-10-26 06:09:11
INFO: restore size = 7.1GB, file total = 995
INFO: restore command end: completed successfully (28382ms)

pgBackRest writes the recovery settings into postgresql.auto.conf and creates recovery.signal for you:

ini
restore_command = 'pgbackrest --stanza=main archive-get %f "%p"'
recovery_target_time = '2026-10-26 00:41:43.351658+00'
recovery_target_inclusive = 'false'
recovery_target_action = 'promote'

Then start PostgreSQL and read its log. These are the lines from our run (times are when recovery ran, five minutes after the accident):

text
00:46:27.325  starting point-in-time recovery to 2026-10-26 00:41:43.351658+00
00:46:27.330  redo starts at 3/CC000028
00:46:35.226  recovery stopping before commit of transaction 12586, time 2026-10-26 00:41:43.351658+00
00:46:35.226  redo done at 4/28699A28 system usage: CPU: user: 1.01 s, system: 0.63 s, elapsed: 7.90 s
00:46:35.226  last completed transaction was at log time 2026-10-26 00:41:43.228894+00
00:46:35.240  selected new timeline ID: 2
00:46:35.341  archive recovery complete
00:46:45.948  database system is ready to accept connections

recovery stopping before commit of transaction 12586 is the line to look for. Replay covered 1.47 GB of WAL (92 segments, from the backup's start to the DELETE) in 7.9 seconds; the other 10.6 seconds were the end-of-recovery checkpoint writing all those replayed pages to disk.

text
step                                  in place (--delta)     into an empty directory
pgBackRest restore (7.1 GB)           28.4 s                 21.1 s
WAL replay (1.47 GB)                  7.9 s                  8.2 s
start to accepting writes             19.5 s                 not timed (same steps)
total, not counting the stop          ~48 s                  about 40 s

The delta restore was the slower one here, because the bad DELETE and three minutes of writes had touched nearly every file of the orders table and pgbench's tables, so it hashed everything and rewrote most of it anyway. Delta pays off when little has changed (a dropped small table, a recent accident on a mostly static database), and in any case it saves you deleting the data directory by hand.

How this scales: the file part is a copy plus zstd decompression, about 250 to 340 MB/s on this SSD, so a 10 GB database is roughly 30 to 40 seconds of restore from a local repository. From object storage the download comes first: about 1.2 GB compressed at our ratio, 10 seconds on a gigabit link, much more on a slower one. Replay depends on how much WAL lies between the backup and the target, not on the database size: a day of WAL at our test's rate is about 130 GB and would replay for something like 12 minutes at the speed we saw. That is the real argument for a full or differential backup every day or two rather than once a week.

Timelines: what changed after the restore

Recovery ended on timeline 2. From here the cluster's WAL segment names start 00000002..., and the archive got a small history file that records where the timelines split:

text
$ cat 00000002.history
1	4/28699A28	before 2026-10-26 00:41:43.351658+00

This is what keeps the two histories apart. The archive still holds timeline 1's WAL after 4/28699A28, the version of events where the DELETE happened and 4,309 more orders came in. Nothing will replay that by accident, because recovery follows the history file, but it is still there if you need to go back and read it.

We restored a second time into an empty directory to time it, with the same target. That run became timeline 3, with a history file of its own: timeline numbers are never reused, so every recovery that promotes gets a fresh one. By default (recovery_target_timeline = 'latest') recovery follows the newest timeline in the archive; pass --target-timeline=current to pgBackRest to stay on the timeline the backup was taken on, which is what you want when the target is before the latest split.

Two chores after any PITR on a production cluster: take a new full backup at once, so the next restore doesn't have to replay across the timeline switch; and check that archiving resumed (pgbackrest check), since the promoted server now archives timeline 2 to the same repository.

What the rewind cost, and the better way to use it

Pilotbase connected to the restored PostgreSQL 18 cluster: 6,003,069 orders, 5,909,287 dated before 1 October, last order id 6003069, last payment at 00:41:43.226883 UTC

The restored database had all 6,003,069 orders back, 5,909,287 of them older than 1 October. It also had nothing that happened after 00:41:43.35: the 4,309 orders and 12,941 payment transactions made during the three minutes before anyone noticed were gone. On a real shop that's worse than a slow afternoon. PITR rewinds the whole cluster; it has no way to undo one statement and keep the rest.

So in production, restore to a separate place and copy back only what was lost. pgBackRest restores into any directory with --pg1-path, and --archive-mode=off stops the copy from pushing its own timeline into your repository:

text
$ sudo -u postgres pgbackrest --stanza=main --pg1-path=/var/lib/postgresql/pitr \
      --type=time "--target=2026-10-26 00:41:43.351658+00" --target-exclusive \
      --target-action=promote --archive-mode=off restore
$ sudo -u postgres /usr/lib/postgresql/18/bin/pg_ctl -D /var/lib/postgresql/pitr -o "-p 5433" start

Production stays up the whole time. When the copy is ready, move the deleted rows across. We measured the first two steps on the test data: the export of 5,909,287 rows to a 923 MB CSV took 3.4 s and loading it into a staging table 7.9 s. (The final INSERT ran out of disk on the test machine, so we have no time for it.)

sql
-- on the restored copy (port 5433)
\copy (SELECT * FROM orders WHERE created_at < '2026-10-01') TO 'deleted_orders.csv' CSV

-- on production
CREATE TABLE orders_restore (LIKE orders);
\copy orders_restore FROM 'deleted_orders.csv' CSV
INSERT INTO orders OVERRIDING SYSTEM VALUE
SELECT * FROM orders_restore
ON CONFLICT (id) DO NOTHING;      -- rows nobody deleted stay as they are

OVERRIDING SYSTEM VALUE is needed because id is GENERATED ALWAYS; it keeps the original IDs, so foreign keys from other tables still point at the right rows. If child tables had ON DELETE CASCADE, their rows went too and need the same treatment, parents first. Check the counts on both sides before you drop the staging table, which is a two-connection job: Pilotbase keeps production and the restored copy open in one window, so the same count query runs against each.

Without pgBackRest: archive_command and pg_basebackup

Everything above works with PostgreSQL's own tools too. The archive_command from the continuous archiving chapter copies each segment and refuses to overwrite one that exists:

ini
archive_command = 'test ! -f /mnt/wal_archive/%f && cp %p /mnt/wal_archive/%f'

$ pg_basebackup -D /backups/base-2026-10-26 -Ft --compress=server-zstd:3 -X stream -c fast

# to restore: unpack base.tar.zst into an empty data directory, then
restore_command = 'cp /mnt/wal_archive/%f %p'
recovery_target_time = '2026-10-26 00:41:43.351658+00'
recovery_target_inclusive = off
recovery_target_action = promote
$ touch $PGDATA/recovery.signal

pg_basebackup was as fast as pgBackRest here (12.9 s against 12.7 s) and the result about the same size. What you give up: WAL is stored uncompressed (16 MB per segment instead of our 5.45 MB average), cp doesn't fsync the file so a crash can leave a half-written segment in the archive, there's no retention, no check, no delta restore, no parallel restore, and no S3. For one small server with a cron job that prunes old backups it is fine; past that, pgBackRest or Barman earn their place.

A cheap S3-compatible repository

A repository on the same disk as the database protects you from a bad DELETE and nothing else. pgBackRest speaks S3 to any compatible store; you can keep the local repository as repo1 and add the bucket as repo2, and archive-push writes to both:

ini
[global]
repo2-type=s3
repo2-path=/pgbackrest
repo2-s3-bucket=db-backups
repo2-s3-endpoint=<account id>.r2.cloudflarestorage.com    # or s3.us-west-004.backblazeb2.com
repo2-s3-region=auto                                        # B2: us-west-004
repo2-s3-key=<access key>
repo2-s3-key-secret=<secret>
repo2-retention-full=2
repo2-cipher-type=aes-256-cbc                               # encrypt before it leaves the server
repo2-cipher-pass=<long random passphrase, kept somewhere other than this server>

What that costs depends mostly on how much WAL you write. A worked example for a 10 GB database with two full backups kept (3.25 GB each, assuming your data compresses like our WAL did, to 32.5%) and 14 days of WAL, at list prices checked on 26 October 2026:

text
WAL per day (raw)          repository   Cloudflare R2   Backblaze B2   Amazon S3 Standard
2 GB   (quiet app)          16 GB       $0.08/month     $0.04/month    $0.36/month
20 GB  (busy app)           98 GB       $1.31/month     $0.61/month    $2.24/month
132 GB (our test, 24 h)    607 GB       $8.96/month     $4.15/month   $13.96/month

R2: $0.015/GB-month, first 10 GB free, no egress fees
B2: $6.95/TB-month, first 10 GB free, egress free up to 3x what you store
S3 Standard (us-east-1): $0.023/GB-month; downloads to the internet $0.09/GB after 100 GB/month

For a restore, egress matters as much as storage: pulling 607 GB out of S3 would cost about $46, while R2 charges nothing and B2 nothing within its 3x allowance. Request charges are small for pgBackRest (one PUT per WAL segment, about 8,000 a day at our test's rate). Prices: Cloudflare R2, Backblaze B2, Amazon S3.

A drill worth an hour a month

text
0. once:                               CREATE TABLE drill_log (at timestamptz, note text);
1. note the time, write a marker row:  INSERT INTO drill_log VALUES (now(), 'before');
2. wait a minute, write another:       INSERT INTO drill_log VALUES (now(), 'after');
3. restore to a spare directory or VM with --pg1-path and --archive-mode=off,
   targeting a time between the two rows
4. start it on another port; the 'before' row must be there and the 'after' row must not
5. write down how long steps 3-4 took: that is your real recovery time

Step 5 is the number to know before the day you need it: the time to restore files from where the repository really is, plus replay of however much WAL has piled up since the last full backup.

Open production and the restored copy side by side, run the same count on each and browse the rows before you copy them back. Pilotbase is free for Windows, macOS and Linux, or Docker on a server.

Get Pilotbase

Exported the restored rows and the live table to CSV? csvDiff shows which rows are missing or changed between the two files, in your browser with nothing uploaded.

Open csvDiff

Related: A pg_dump you have never restored is not a backup · Upgrading Postgres 16 to 18 with logical replication

Sources: PostgreSQL 18 documentation: recovery target settings (inclusive default on, numeric offsets, timeline 'latest' default, promote/pause/shutdown), continuous archiving and PITR (archive_command example, recovery.signal, timelines and history files), pg_waldump. pgBackRest user guide and configuration reference (stanza-create, check, --delta, --type=time with --target-exclusive, --pg1-path, --archive-mode, S3 repositories, repository encryption). Storage prices from the providers' pricing pages, 26 October 2026. Test: PostgreSQL 18.6 and pgBackRest 2.60.0 on macOS, Apple silicon Mac mini (10 cores, 16 GB), repository on the internal SSD, 26 October 2026.