Restoring production in 10 seconds

Most backup systems report one thing: the job exited zero. That tells you a program finished. It doesn't tell you whether you can get your business back. The Sin fleet runs on one rule instead: a backup is not a backup until it has been restored, meaning file count, byte count, hash verification, a real test extraction and timings. An exit code proves nothing.

This is the story of the first time we held production to that rule.

The production web server hosts live sites, mail and two Postgres clusters. It's run under a strict read-only mandate: the fleet may inspect it but may not write a byte to it. No packages, no SSH keys added, no agents installed.

That constraint shaped everything. The vault can't pull from production, because that would need a key installed on production. So the hub streams a single tar archive out of production and into the vault, and restic snapshots it there, encrypted and deduplicated.

• Snapshot: 40,282 files, 2.365 GiB processed, 1.273 GiB stored after dedup and compression.
• Covered: every site tree, all mailboxes, the hosting panel's configuration, the web and mail server configs, a consistent MariaDB dump and the raw Postgres data volumes.
• Deliberately excluded: 1.41 GB of node_modules and cache. It was measured rather than guessed, and it exactly accounts for the gap between source and staged size.
• Afterwards every service on production still showed its pre-backup start time. Nothing was touched.

The snapshot was restored onto a separate, expendable drill host that had never held production data, then checked against the live originals.

• Restored 48,404 files and directories (2.365 GiB) in 10 seconds, 13 seconds wall clock.
• Hashed 25 files against live production: 0 mismatches.
• Loaded the MariaDB dump into a throwaway container. Every database opened, the largest with 169 tables.
• Opened both Postgres clusters and ran real COUNT(*) queries. The application database held 65 tables and 1,678 rows.
• Copied the snapshot to a second repository on separate hardware and verified it clean.
• Wiped the drill target afterwards, because the restored tree contains mail and credentials.

Docker volumes keep their data one level down, in a _data directory. Point Postgres at the volume itself instead of <volume>/_data and it doesn't fail. It happily initialises a brand-new empty cluster. Everything starts and every health check is green, and there's no data.

A backup report would never catch that. The only reason we found it is that the drill actually queried the database.

The two Postgres clusters turned out to be different major versions, 15 and 16. A restore runbook that assumes one version restores one database and fails the other with "database files are incompatible". Under pressure, at 3 a.m., that's how you lose a day.

The fix is written into the runbook now: match the image to the data. The PG_VERSION file in the data directory is authoritative.

• Row counts from pg_stat_user_tables read zero on a freshly started container, which makes a full database look empty. Only COUNT(*) counts.
• restic check --read-data-subset=1/50 on a repository this size selects zero data packs. It verifies structure, not bytes. The full --read-data pass reads all 107 packs (1.6 GiB) in about 4 seconds, so there's no reason to ever quote a subset as "verified".

An earlier drill of the hub's own backup worked perfectly. It restored 49,157 files in 5 seconds, and the hashes matched. The receipt noted the temporary password file being shredded. It said nothing about the restored tree itself, which sat on the drill host for almost 13 hours with a copy of secrets in it.

The fix went into the script, not a sticky note. The drill now shreds secret-bearing files, removes the target, verifies it's gone and prints that line in its receipt. A gap survives a receipt when the receipt only lists what went right.

Every failure above looked like success from the outside. The empty cluster started, the subset check passed and the drill reported clean. The only defence is to finish the job: restore it, open it, count it and hash it, then write down exactly what you proved and nothing more.

That's the standard I hold the fleet to, and the one I bring to client systems.