I’ve spent the last few weeks writing down every way I could find that a backup job can be green and the restore still fails. Ended up with eleven distinct failure modes, each with why nothing notices it and what it actually takes to detect:
The one I’d most like this forum’s opinion on is #4, because it’s restic-specific and I only half-trust my own understanding of it: restic caches tree and index packs locally. With a warm cache, a repository whose tree pack is corrupt can still restore cleanly, because the damaged pack never gets read. So on the machine that took the backup — where the cache is warm — corruption can be invisible even to a real restore, unless caching is explicitly disabled. If that’s wrong or overstated, I’d rather be corrected here than have it sit in a document people read. The other one I think is underrated is #8, cross-component referential breakage: the database dumped at 02:00, the blob store synced at 03:00, and now the DB references files that don’t exist. Both backups are individually perfect. Nextcloud, Immich, Paperless — anything where a database and an object store have to agree. Almost nothing checks it. The taxonomy is the part I actually want feedback on. There’s also a tool I built against it — constat, a Go binary that restores a snapshot into a temp dir or a disposable Postgres container, runs assertions against the
result, and emits a signed dated report:
Fair warnings: it’s v0, restic is the only backend, Postgres is the only database, and the codebase was largely written by Claude with me directing and reviewing rather than typing — that’s stated up front in the README and I’d rather say it here than have someone find it. Treat it as experimental. The taxonomy stands on its own regardless. Corrections to the list are the most useful thing anyone could give me.
Regarding #4:restic check uses a separate cache folder to detect server-side data corruption. That is, if restic check reports no error, then the “intact in local cache, but broken in repository” case can’t occur. (Btw, index files can be rebuilt from scratch)
For cloud backends like S3 or rest-server, restic includes checksums in the upload that are verified by the server. This completely rules out data corruption during uploads to those backends. While corruption at rest could still occur, the cloud backends typically employ redundant storage to also rule that out.
For all backends except rclone (and some uncommon filesystems for local backups), somewhat recent restic versions can guarantee that a backup run only succeeds if every file has been completely uploaded and written to storage. The basic approach here is that it waits until the OS or backend confirms that a file was successfully stored. That is, “truncated uploads” are largely a no show for restic.
Your description is missing one not entirely uncommon cause of bit rot: corruption in CPU or RAM directly while the backup runs! These cases are extremely hard to detect as the only way is to fully diff the backup against the original data. Restic decodes and verifies the checksums of file chunks before uploads, which partially mitigates this. There are also several layers of checksums such that data corruption is often not silent.
Regarding #5: I’ve seen the “passphrase exists only on the backed up system” case once in person (not my backup though). Luckily, the system in question was just a few days down for maintenance and not entirely gone.
Regarding #7: a missing executable bit rather sounds like a restore tool bug. But file ownership / SELinux are indeed somewhat relevant. Although with modern container-based setups, this is also less of a problem as there’s usually just one user for a volume.
Regarding #8: If blob stores are append-only, then a newer blob store and an older database are fine. You just end up with some unreferenced data.
#9 sounds like a problem that should be solved using Infrastructure-as-Code. It should contain the postgres version that was in use at the time of the backup. Then the runbook can just reference this. But that won’t solve the general problem of “runbook bitrot”. That can only be solved by periodically testing the restore process. (I know that’s tedious such that larger companies often have corresponding compliance controls)
Edit:
“Cross-component referential breakage” should IMHO also be ordered earlier in the list as this is a problem at backup time (#1-#3), not so much storage (#4-#6) or restore (#7-#11 without #8) time. The distinction matters as restore time errors might be solvable later on at the cost of a longer restore time. But backup or storage time issues are permanent.
Edit 2:
Regarding #4: For the cloud backends a restic check without --read-data already verifies nearly everything except for the backup-time bitrot. (Assuming the cloud backend does not corrupt the data after verifying the checksum during upload). The main protection here is, however, to follow the 3-2-1 rule (or one of its variants): 3 independent data copies (original + 2 backups), 2 different storage media and one offsite. This drastically increases the chance that bitrot only affects one and not both backups at the same time.
Thank you for the detailed read, — good catches on both counts.
On #9 (runbook bitrot): Pinning the PG version via IaC is the right fix for the specific “does this restore command still match reality” failure, and I’ll fold that into the writeup — the runbook should reference the version that was live at backup time, not whatever’s current. That said, IaC only prevents the drift; it doesn’t catch it after the fact if the runbook was written before IaC was in place. That’s the gap constat is aimed at: it doesn’t trust the runbook, it just does the restore and asserts on the result, so version drift shows up as a failed assertion instead of a surprise at 3am. Periodic restore testing being “enforced by compliance in larger orgs” and basically ignored everywhere else is exactly the gap I built this for — wanted something cheap enough to run on a cron even without a compliance mandate forcing it.
On reordering #8 (cross-component referential breakage): Agreed, and it’s a fair criticism of the taxonomy’s structure — that’s a backup-time consistency failure (the DB dump and blob sync being taken at different instants), not a restore-time one, and conflating the two blurs a useful distinction: backup-time corruption is often permanent by the time you notice, restore-time errors are at least recoverable if you catch them early. I’ll move it up in the next revision. Curious whether you’d group it near #1 (silent non-execution) or as its own early category, since the failure mode is “backup succeeded per-component but the components disagree,” not “backup didn’t happen.”
On #4 (bitrot): Right that restic check without --read-data gives real coverage on cloud backends — it verifies pack existence and tree/index consistency, but it doesn’t read pack contents back and compare against the recorded hash, so a bit-flipped-but-present pack can still pass. That’s the case I was pointing at: warm cache or a check that stops short of --read-data/--read-data-subset can both let a corrupted pack go undetected until an actual restore touches it. Agreed 3-2-1 is the right mitigation for surviving corruption once it happens; I’d frame it as complementary rather than overlapping with --read-data — one lowers the odds correlated corruption wipes every copy, the other is what actually tells you corruption happened in the first place. A full --read-data run is expensive enough that most people don’t do it routinely, which is part of why constat does an actual restore-and-verify instead of relying on check alone.