load indexes
[0:02] 100.00% 12 / 12 index files loaded
check all packs
check snapshots, trees and blobs
[0:17] 100.00% 14 / 14 snapshots
read all data
pack c2ac6a60*** contains 1 errors: [blob 80a9f0bc***: read blob <data/80a9f0bc> from c2ac6a60: wrong data returned, hash is ea4d34a4be***]
pack 53120b13*** contains 1 errors: [blob c61f3e5b***: read blob <data/c61f3e5b> from 53120b13: wrong data returned, hash is bc191d5522***]
pack 3e3d9cb9*** contains 1 errors: [blob 67581c07***: read blob <data/67581c07> from 3e3d9cb9: wrong data returned, hash is 2a3b56b4af***]
[3:56:54] 100.00% 42897 / 42897 packs
The repository contains pack files with damaged blobs. These blobs must be removed to repair the repository. This can be done using the following commands:
RESTIC_FEATURES=repair-packs-v1 restic repair packs c2ac6a60*** 53120b13*** 3e3d9cb9***
restic repair snapshots --forget
Corrupted blobs are either caused by hardware problems or bugs in restic. Please open an issue at https://github.com/restic/restic/issues/new/choose for further troubleshooting!
Fatal: repository contains errors
restic version
restic 0.16.3 compiled with go1.21.6 on windows/amd64
DiskGenius Disk Verify Result for the parition where the repository is: 0 bad sector areas found.
The output from the check command is the proper way forward. Although, after running restic repair packs you can alternatively continue with Step 4 of the Troubleshooting documentation.
Did you use restic 0.16.0 (which had a bug that could cause this kind of data corruption)?
I recently began having the same problem when using restic 0.16.3
load indexes
[0:00] 100.00% 1 / 1 index files loaded
check all packs
check snapshots, trees and blobs
[0:00] 100.00% 1 / 1 snapshots
read all data
pack 9001cfce482d26b5edb5a29654cce29f82455a1685b0c36410f05fa52825842e contains 3 errors: [blob 3ffae4b1a2315b80506e5cf60446b122b4ce308f012cecb44f0289cc6ebe5df8: read blob <data/3ffae4b1> from 9001cfce: wrong data returned, hash is d555423aba549a09167630a77fb597d63f72f312262ce74466405b9be60b1ace blob 69d3b597c7299368e40412fcaf92566974a137229ff7e2b4eb90d1465f33ae0b: read blob <data/69d3b597> from 9001cfce: wrong data returned, hash is c9df092f6dc7c439421f583ddd8d0118aa3af13c64ed91b4ce9f2379ec8b59ef blob c320093a9872b1b8e563915a4b8e3db07b0ecff55ede50dc70c3470a0392de03: read blob <data/c320093a> from 9001cfce: wrong data returned, hash is 8623e5425e6045bd42ac546f1c79692c23e088364e300fe765b09ef3b557a09a]
pack a74dc87dfb62b77f6afed7d867c09ae5b6936fbecd5bb3f26158f6c74a65d524 contains 1 errors: [blob e6250b5955cb41a212e569f9914ceaf790006d3b8c422c8137b7a53a9fb7576e: read blob <data/e6250b59> from a74dc87d: wrong data returned, hash is e32eace92b64ec5c9aa0ea1507f9281a737e4a81450310eab38a1b0cc619aa4c]
pack 1d687c094b382b5c7c0564f9a52944742fd703a773d169e1be07eef6dc256d46 contains 1 errors: [blob 201884b3bd529279917a8b64721785815c312f06855a38e6ed4337d84e1accc8: read blob <data/201884b3> from 1d687c09: wrong data returned, hash is 6f3045708e6b558b617cee51cd07c5ef625074cf2d0034340ca6eb528e4ab8f0]
[0:54] 100.00% 1655 / 1655 packs
The repository contains pack files with damaged blobs. These blobs must be removed to repair the repository. This can be done using the following commands:
RESTIC_FEATURES=repair-packs-v1 restic repair packs 9001cfce482d26b5edb5a29654cce29f82455a1685b0c36410f05fa52825842e a74dc87dfb62b77f6afed7d867c09ae5b6936fbecd5bb3f26158f6c74a65d524 1d687c094b382b5c7c0564f9a52944742fd703a773d169e1be07eef6dc256d46
restic repair snapshots --forget
Corrupted blobs are either caused by hardware problems or bugs in restic. Please open an issue at https://github.com/restic/restic/issues/new/choose for further troubleshooting!
Fatal: repository contains errors
restic 0.16.3 compiled with go1.21.6 on linux/amd64
I can reproduce the problem at will using a test repository with a single snapshot backing up a single file. Check operations on numerous other repositories succeed with no errors. As strange as it seems, there appears to be some characteristic of this particular file that results in the error.
Dropping back to 0.16.2 resolves the issue, at least in my case.
I remembered the issue but had understood from the 0.16.1 changelog that it had been addressed in that release. Apparently there was/is more to it than that.
Thanks for all the work you do on restic! It is much appreciated.
The original issue in 0.16.0 has been fixed, but another, new bug was introduced in a later version of the compression library. And unfortunately restic 0.16.3 uses that version.
First: Is there a way to find out which snapshots/files are effected?
For this second one, I just want to check my understanding of the process that would heal the repo.
The man page states:
The “repair packs” command extracts intact blobs from the specified pack files, rebuilds the index to remove the damaged pack files and removes the pack files from the repository.
So, the operation repacks the good blobs, deletes the old pack file they were in, and rewrites the index so that it contains the new, good pack file instead of the old, damaged one. It does not seem to touch snapshots. Since rerunning backups helps, I’m guessing snapshots refrence blobs, not packs. So when a new backup is made containing the data associated with the removed blob(s), they now exist in the repo again, and thus making the reference valid again.
To get the list before running restic repair snapshots you can use restic find --blob <blob ids> and pass the list of damaged blobs ids to it. Based on the check log from underhillian that would be restic find --blob 3ffae4b1a2315b80506e5cf60446b122b4ce308f012cecb44f0289cc6ebe5df8 69d3b597c7299368e40412fcaf92566974a137229ff7e2b4eb90d1465f33ae0b c320093a9872b1b8e563915a4b8e3db07b0ecff55ede50dc70c3470a0392de03 e32eace92b64ec5c9aa0ea1507f9281a737e4a81450310eab38a1b0cc619aa4c 6f3045708e6b558b617cee51cd07c5ef625074cf2d0034340ca6eb528e4ab8f0.
restic repair snapshots --forget also prints which files/snapshots were affected.
Exactly.
The mapping between blob ids and pack files is stored in the repository index. So snapshots only store blob ids. That allows later backups to readd missing blobs.
restic repair packs rebuilds the pack files, restic repair snapshots afterwards modifies the snapshots to only reference blobs that still exist in some pack file.
Thanks for confirming; data corruption is scary and understanding what’s going on helps.
One point of clarity: so long as I don’t run restic repair snapshots --forget, and so long as I can backup good copies of files which “contain” the missing/corrupted blobs, I shouldn’t need to worry about data loss, correct? And setting --dry-run with that will let me find out where the damage is without touching anything.
I’ve made snapshots of the entire repo, so a mistake on my part shouldn’t cause permanent data loss, but again, I want to make sure I know what I’m doing. (Speaking of Btrfs, I scrubbed the volume and it came back okay.)
Yes, that variant should allow you to repair the damaged blobs.
It’s faster and more thorough to run restic check (ideally including the --read-data option). Although restic repair snapshots --dry-run also reports missing blobs. Only check --read-data is able to detect corrupted blobs.
