I’ve spent the last few weeks writing down every way I could find that a backup job can be green and the restore still fails. Ended up with eleven distinct failure modes, each with why nothing notices it and what it actually takes to detect:
The one I’d most like this forum’s opinion on is #4, because it’s restic-specific and I only half-trust my own understanding of it: restic caches tree and index packs locally. With a warm cache, a repository whose tree pack is corrupt can still restore cleanly, because the damaged pack never gets read. So on the machine that took the backup — where the cache is warm — corruption can be invisible even to a real restore, unless caching is explicitly disabled. If that’s wrong or overstated, I’d rather be corrected here than have it sit in a document people read. The other one I think is underrated is #8, cross-component referential breakage: the database dumped at 02:00, the blob store synced at 03:00, and now the DB references files that don’t exist. Both backups are individually perfect. Nextcloud, Immich, Paperless — anything where a database and an object store have to agree. Almost nothing checks it. The taxonomy is the part I actually want feedback on. There’s also a tool I built against it — constat, a Go binary that restores a snapshot into a temp dir or a disposable Postgres container, runs assertions against the
result, and emits a signed dated report:
Fair warnings: it’s v0, restic is the only backend, Postgres is the only database, and the codebase was largely written by Claude with me directing and reviewing rather than typing — that’s stated up front in the README and I’d rather say it here than have someone find it. Treat it as experimental. The taxonomy stands on its own regardless. Corrections to the list are the most useful thing anyone could give me.
Regarding #4:restic check uses a separate cache folder to detect server-side data corruption. That is, if restic check reports no error, then the “intact in local cache, but broken in repository” case can’t occur. (Btw, index files can be rebuilt from scratch)
For cloud backends like S3 or rest-server, restic includes checksums in the upload that are verified by the server. This completely rules out data corruption during uploads to those backends. While corruption at rest could still occur, the cloud backends typically employ redundant storage to also rule that out.
For all backends except rclone (and some uncommon filesystems for local backups), somewhat recent restic versions can guarantee that a backup run only succeeds if every file has been completely uploaded and written to storage. The basic approach here is that it waits until the OS or backend confirms that a file was successfully stored. That is, “truncated uploads” are largely a no show for restic.
Your description is missing one not entirely uncommon cause of bit rot: corruption in CPU or RAM directly while the backup runs! These cases are extremely hard to detect as the only way is to fully diff the backup against the original data. Restic decodes and verifies the checksums of file chunks before uploads, which partially mitigates this. There are also several layers of checksums such that data corruption is often not silent.
Regarding #5: I’ve seen the “passphrase exists only on the backed up system” case once in person (not my backup though). Luckily, the system in question was just a few days down for maintenance and not entirely gone.
Regarding #7: a missing executable bit rather sounds like a restore tool bug. But file ownership / SELinux are indeed somewhat relevant. Although with modern container-based setups, this is also less of a problem as there’s usually just one user for a volume.
Regarding #8: If blob stores are append-only, then a newer blob store and an older database are fine. You just end up with some unreferenced data.
#9 sounds like a problem that should be solved using Infrastructure-as-Code. It should contain the postgres version that was in use at the time of the backup. Then the runbook can just reference this. But that won’t solve the general problem of “runbook bitrot”. That can only be solved by periodically testing the restore process. (I know that’s tedious such that larger companies often have corresponding compliance controls)
Edit:
“Cross-component referential breakage” should IMHO also be ordered earlier in the list as this is a problem at backup time (#1-#3), not so much storage (#4-#6) or restore (#7-#11 without #8) time. The distinction matters as restore time errors might be solvable later on at the cost of a longer restore time. But backup or storage time issues are permanent.
Edit 2:
Regarding #4: For the cloud backends a restic check without --read-data already verifies nearly everything except for the backup-time bitrot. (Assuming the cloud backend does not corrupt the data after verifying the checksum during upload). The main protection here is, however, to follow the 3-2-1 rule (or one of its variants): 3 independent data copies (original + 2 backups), 2 different storage media and one offsite. This drastically increases the chance that bitrot only affects one and not both backups at the same time.
Thank you for the detailed read, — good catches on both counts.
On #9 (runbook bitrot): Pinning the PG version via IaC is the right fix for the specific “does this restore command still match reality” failure, and I’ll fold that into the writeup — the runbook should reference the version that was live at backup time, not whatever’s current. That said, IaC only prevents the drift; it doesn’t catch it after the fact if the runbook was written before IaC was in place. That’s the gap constat is aimed at: it doesn’t trust the runbook, it just does the restore and asserts on the result, so version drift shows up as a failed assertion instead of a surprise at 3am. Periodic restore testing being “enforced by compliance in larger orgs” and basically ignored everywhere else is exactly the gap I built this for — wanted something cheap enough to run on a cron even without a compliance mandate forcing it.
On reordering #8 (cross-component referential breakage): Agreed, and it’s a fair criticism of the taxonomy’s structure — that’s a backup-time consistency failure (the DB dump and blob sync being taken at different instants), not a restore-time one, and conflating the two blurs a useful distinction: backup-time corruption is often permanent by the time you notice, restore-time errors are at least recoverable if you catch them early. I’ll move it up in the next revision. Curious whether you’d group it near #1 (silent non-execution) or as its own early category, since the failure mode is “backup succeeded per-component but the components disagree,” not “backup didn’t happen.”
On #4 (bitrot): Right that restic check without --read-data gives real coverage on cloud backends — it verifies pack existence and tree/index consistency, but it doesn’t read pack contents back and compare against the recorded hash, so a bit-flipped-but-present pack can still pass. That’s the case I was pointing at: warm cache or a check that stops short of --read-data/--read-data-subset can both let a corrupted pack go undetected until an actual restore touches it. Agreed 3-2-1 is the right mitigation for surviving corruption once it happens; I’d frame it as complementary rather than overlapping with --read-data — one lowers the odds correlated corruption wipes every copy, the other is what actually tells you corruption happened in the first place. A full --read-data run is expensive enough that most people don’t do it routinely, which is part of why constat does an actual restore-and-verify instead of relying on check alone.
On #4. Some cloud providers have “free” (meaning, “included in the price”) download quotas. For example, I use Backblaze B2 for my DR backup with restic. B2 gives you a free monthly download quota of 3x your monthly average storage. So, if you store 1TB, you get 3TB of free download.
This lets me use --read-data-subset=n/t, where I set t to 30 days and I increment n once a day. My thinking is that will cover the entire repo over the span of 30 days, but keep the downloads well within the quota. Does that sound like a reasonable assumption?
Thanks for writing the document. I found it very useful to mull over.
That’s a solid approach — the math works out with room to spare. --read-data-subset=n/t splits pack files into t roughly-equal buckets by hash, so cycling n from 1 to 30 over a month reads close to 100% of your repo. That’s about 1x your repo size in downloads, not 3x, so you’re comfortably inside quota.
Two things to watch:
n is 1-indexed (1 to t) — make sure your script wraps correctly at day 31.
B2 also caps free “Class B” download transactions (API calls), separate from bytes. If your repo has lots of small pack files, that’s worth checking too.
Glad the doc was useful to think through!
The way we do that is to just have a job that rolls a dice on which backup to restore and selects a bunch of files, if there is data corruption the restic will just fail during the restore. It being random gives better chance on hitting some corrupted blocks
My take is: the tool is never perfect so you want test that actually restores the backup either way. Only thing restic-wise I miss is ability to stream to tarball - that aside from some other utility (if someone else needs old version of a dir I could just directly make a tarball instead of requiring intermediate space) also allows for easy “do full restore, pipe to dev null”
As a former Database Administrator I have seen or heard of many backups which fail to restore for various reasons. I will attempt to list some issues which I have about your thoughts:
Why do you say the report is signed? Who signed it? What happens if they don’t sign it? Is it ment to be an overall status of the backup systems? If people are ignoring jobs that are failing, why won’t they also ignore this report?
One of the first backups that I heard of that failed to restore did a full backup (all files) then only incremental backups (changed files or blocks). When they went to do the restore, somehow the full backup did not exist anymore so the incrementals could not be applied. This is not a problem with restic because there is no separate full backup. In some sense all of the backups are incrementals.
The example of cross-component breakage is confusing to me. I would reword the example as two financial systems which need to be backed up. If are separate each would be fine. But if the financial systems are dependant / coordinated with each other then the backups and restores must be coordinated because otherwise they will become out of sync and financially incorrect.
I have seen backup scripts which have backups coded very specifically. This failed because the lists were updated manually and people forget to add the new file, or the file that doesn’t exactly match the template syntax. I much prefer to have dynamically generated lists which are automatically updated. My preference is also to have dynamically generated restore scripts so the person who has been awakened at 3 am does not have a large cognitive work load to do a restore.
Properly backing up a database needs to be organized by the vendor of the database. Each database vendor has there own requirements. Some will supply software and others will only supply documentation.
A backup / restore is dealing with computers and data but the purpose is actually to help people. Your list does not seem to document the requirements to deal with people. For example who has the authority to require / demand a restore? Can a janitor require that the whole email system be restore because a email was lost? If the main company financial system has to be restored, should the Chief Financial Officer be required to sign that off? I had a computer project manager say that certain database changes could be done. When I asked what happens if it does not go perfectly he said that a restore would be done. When I told him a restore would take more than 24 hours and the user would not have a functioning system during that time, I told him to give me in writing that this was ok by the main business manager.
As you likely know, RAID is not backup. If something is deleted it is still gone even if was on file system on RAID. If you have a disk failure on a raid system, replacing the drive will take quite a while before it is back to normal. This can easily be >24 hours.
Do you actually have the spare disk space to do a restore? This is only possible if you are using only 50% of your current disks. You will be able to restore a single file from a restic backup into a temporary directory but you will not have the space to do a full restore.
More people stuff. If a restore is done it needs to be documented who must be informed. For example if a financial server is restored does the application which is on the desktop need to restored in a coordinated way? Who would do that or will it happen automatically? (For some reason I seem to think that PeopleSoft software would check the database then look for the correct version of the application to run but I am not 100% sure.)
More people stuff: A user created “informal” project did not “go into production” formally so it never had a formal, tested backup. It was not on the list of production things we, the people responsible for production, were responsible for looking after. Because of all the informality the system fell through the cracks. I expected to be fired when because something went wrong and there was no backup. (It turned out ok, because the database rolled back on it’s own to a stable state. Backups were then scheduled.)
Some software requires specific license keys. One application I knew about would check something like once every 24 hours to the vendor’s site to see if the license key was valid. The key had some sort of dependency on the cpu / disks so a restore on backup site / backup disks would be invalid. However the vendor said that the app would run fine except there would be a message on the screen for every user to see regarding licensing. Temporary / Emergency licenses were available by contacting the vendor so that worked out fine.
More people stuff: Who is “on call” and who has to pay for the time it takes the people to do the restore? I told a manager when he told me to do a restore that I would not because I had offered to start on Friday afternoon and had been told we don’t need that. But now it was in the middle of the weekend and I did not want to take the travel time to get to the office, do the work, then travel back home. (That job literally nearly killed me: after leaving work late one time I was so tired that I drove on the wrong side of the road and refused to believe I was wrong for a couple of blocks in the dark.)