How we back up — and how we prove the backup works
A backup nobody has read back is not a backup, it is a hope. Here is the order we do things in, and why that order is the whole point.
Apex Operations · 16 October 2026 · 5 min read
Most backup failures are not dramatic. Nobody deletes the backups. What happens is that they quietly stop working, and nobody notices for three weeks, because the only thing being monitored is whether the job ran.
The order matters
Every workspace is dumped nightly. Then, before the row is written that says a backup exists:
- The dump is read back with
pg_restore --list. If it does not contain a schema, the file is deleted and nothing is recorded as a backup. - It is checksummed and sized. An empty file is not a backup either.
- Only then is the set recorded.
The same discipline applies when a customer leaves: the dump is taken, read back and checksummed before the live database is dropped. Anything else is a one-way door with a hope attached.
Failures are recorded as failures
A backup log containing only successes cannot tell you that backups stopped working. Ours records the failed attempt, with the reason, and a workspace whose most recent attempt failed is reported as not covered — on the dashboard, and by a command whose exit code makes it usable as a monitor.
Verification is a separate run
Taking a backup and verifying it in the same pass only ever confirms the dump you just wrote. That proves nothing about the ones you would actually rely on. So verification runs hours later, re-reads every dump on disk, and marks any set whose checksum no longer matches as failed — because an operator reading the list must not be told a corrupt file is a backup.
Restoring takes a safety copy first
A restore to the wrong point in time is a mistake. A restore that also destroys the state it replaced is an incident. So before any restore, the current contents are dumped and verified — and if that safety copy fails, the restore does not happen.
Pruning has a floor
Retention removes sets past their expiry, but always keeps the most recent few whatever their age. If backups have been failing for a month, the last good copy must not be pruned on schedule — which is precisely when it would be.
Engineering / 1 Oct 2026 / 6 min
Why every customer gets their own database
Most SaaS products put every customer in one set of tables behind a tenant column. We give each workspace its own PostgreSQL database instead. Here is what that costs us, and what it buys you.
Read it