Repo corruption

Getting Help
1

Hmm I’ve got an issue:

repository 6a4c65bd opened successfully, password is correct
created new cache in /var/folders/78/z94fqn6944l3mcz4mxrwf6ycx3x4jt/T/restic-check
-cache-885551846
create exclusive lock for repository
load indexes
check all packs
pack 1c9d6029: not referenced in any index
pack f1a944f5: not referenced in any index
pack 74232f71: not referenced in any index
pack 60a4c0ca: not referenced in any index
pack 11cdd801: not referenced in any index
5 additional files were found in the repo, which likely contain duplicate data.
You can run restic prune to correct this.
check snapshots, trees and blobs

error for tree 338f199a:snapshots
tree 338f199a: file “209-102.146666.bam” blob 5f922a1838cf3901a3f8a909b4ba5d192a
c3887fd12ca754aa5fa1eed4fa5e53 not found in index
tree 338f199a: file “209-102.146666.bam” blob 3e891f469f4a4387fd3cf6c56e3512c529
50e2d59122002d225ea8ca3b2ba3c2 not found in index
tree 338f199a: file “209-201.146667.bam” blob eae5353c6f78a4168f3e21e6f9c6c24b4b
013252a80b4b663e4b62a04f932f4c not found in index
tree 338f199a: file “209-201.146667.bam” blob 27d6ec01ed2e7dcddc518b402d05df9297
8e619b5a10e504e25ddfdce6f6cbcc not found in index
error for tree 53a5a1e5:snapshots
tree 53a5a1e5: file “209-102.146666.bam” blob 5f922a1838cf3901a3f8a909b4ba5d192a
c3887fd12ca754aa5fa1eed4fa5e53 not found in index
tree 53a5a1e5: file “209-102.146666.bam” blob 3e891f469f4a4387fd3cf6c56e3512c529
50e2d59122002d225ea8ca3b2ba3c2 not found in index
tree 53a5a1e5: file “209-201.146667.bam” blob eae5353c6f78a4168f3e21e6f9c6c24b4b
013252a80b4b663e4b62a04f932f4c not found in index
tree 53a5a1e5: file “209-201.146667.bam” blob 27d6ec01ed2e7dcddc518b402d05df9297
8e619b5a10e504e25ddfdce6f6cbcc not found in index

error for tree bc1b8c1a: snapshots
tree bc1b8c1a: file “209-102.146666.bam” blob 5f922a1838cf3901a3f8a909b4ba5d192a
c3887fd12ca754aa5fa1eed4fa5e53 not found in index
tree bc1b8c1a: file “209-102.146666.bam” blob 3e891f469f4a4387fd3cf6c56e3512c529
50e2d59122002d225ea8ca3b2ba3c2 not found in index
tree bc1b8c1a: file “209-201.146667.bam” blob eae5353c6f78a4168f3e21e6f9c6c24b4b
013252a80b4b663e4b62a04f932f4c not found in index
tree bc1b8c1a: file “209-201.146667.bam” blob 27d6ec01ed2e7dcddc518b402d05df9297
8e619b5a10e504e25ddfdce6f6cbcc not found in index
error for tree dfb3f07a: snapshots
tree dfb3f07a: file “209-102.146666.bam” blob 5f922a1838cf3901a3f8a909b4ba5d192a
c3887fd12ca754aa5fa1eed4fa5e53 not found in index
tree dfb3f07a: file “209-102.146666.bam” blob 3e891f469f4a4387fd3cf6c56e3512c529
50e2d59122002d225ea8ca3b2ba3c2 not found in index
tree dfb3f07a: file “209-201.146667.bam” blob eae5353c6f78a4168f3e21e6f9c6c24b4b
013252a80b4b663e4b62a04f932f4c not found in index
tree dfb3f07a: file “209-201.146667.bam” blob 27d6ec01ed2e7dcddc518b402d05df9297
8e619b5a10e504e25ddfdce6f6cbcc not found in index

error for tree 4399bee8: snapshots
tree 4399bee8: file “209-102.146666.bam” blob 5f922a1838cf3901a3f8a909b4ba5d192a
c3887fd12ca754aa5fa1eed4fa5e53 not found in index
tree 4399bee8: file “209-102.146666.bam” blob 3e891f469f4a4387fd3cf6c56e3512c529
50e2d59122002d225ea8ca3b2ba3c2 not found in index
tree 4399bee8: file “209-201.146667.bam” blob eae5353c6f78a4168f3e21e6f9c6c24b4b
013252a80b4b663e4b62a04f932f4c not found in index
tree 4399bee8: file “209-201.146667.bam” blob 27d6ec01ed2e7dcddc518b402d05df9297
8e619b5a10e504e25ddfdce6f6cbcc not found in index
error for tree 08970217: snapshots
tree 08970217: file “209-102.146666.bam” blob 5f922a1838cf3901a3f8a909b4ba5d192a
c3887fd12ca754aa5fa1eed4fa5e53 not found in index
tree 08970217: file “209-102.146666.bam” blob 3e891f469f4a4387fd3cf6c56e3512c529
50e2d59122002d225ea8ca3b2ba3c2 not found in index
tree 08970217: file “209-201.146667.bam” blob eae5353c6f78a4168f3e21e6f9c6c24b4b
013252a80b4b663e4b62a04f932f4c not found in index
tree 08970217: file “209-201.146667.bam” blob 27d6ec01ed2e7dcddc518b402d05df9297
8e619b5a10e504e25ddfdce6f6cbcc not found in index

[9:05] 100.00% 262 / 262 snapshots
Fatal: repository contains errors

If I do:

restic find --blob 5f922a1838cf3901a3f8a909b4ba5d192ac3887fd12ca754aa5fa1eed4fa5e53 --blob 3e891f469f4a4387fd3cf6c56e3512c52950e2d59122002d225ea8ca3b2ba3c2 --blob eae5353c6f78a4168f3e21e6f9c6c24b4b013252ac80b4b663e4b62a04f932f4c --blob 27d6ec01ed2e7dcddc518b402d05df92978e619b5a10e504e25ddfdce6f6cbcc

Will that be enough to find the issue and then correct it? Thanks!

2

I figured out the snapshots from the above command, and re-ran the backup on just the files that were missing - but for some reason that didn’t fix it even though the files had not been modified.

It was no big deal to forget the offending snapshots, NOT prune, and just back up again fresh. Seems to be working now. Leaving this for others to see. Thanks!

3

I’ve seen running rebuild-index and then running a backup again suggested around here before, e.g. by @alexweiss. Perhaps try that next time :slight_smile:

4

rebuild-index is always suggested if something is wrong with your repo. It ensures that the index correctly reflects the contents of the pack files. And as many parts in restic rely on the the index, this is a crucial step - even more if you had to manually remove some pack files.

In the case of @akabru some blobs were already missing in the index. If those blobs had been still available in pack files but only the index wrong for some reason (e.g. an index file was deleted), rebuild-index alone would have fixed this. Another healing method would be to run another backup which contains exactly the missing data and then re-uploads the missing blobs and adds them to the index.

Actually I doubt that the files were exactly the same as in the snapshots reported by check , see my explanation above…
Or something else went bad in your case :unamused:

3 Likes
5

Oh hey, actually I did do rebuild-index once, before one more “check”. It resolved a little bit but not everything. There were actually two blob files I could not copy at all when moving my repo (guess the drive was dying after all). So I knew there’d be errors. Weird thing is when I did “backup” from an unchanged file set, it didn’t add those blobs back.

Because it wasn’t too big of a deal, I just removed all the snapshots referencing it, didn’t prune (so it wouldn’t have to copy all the duplicate data over the wire) then ran a fresh “backup” and it’s all good now.

This was a dataset from like 2018 that I was “archiving” into the repo to save space (lots of dupes). Hadn’t modified anything. But perhaps some little metadata changed while the drive was mounted. Who knows. :man_shrugging:t2:

6

Scary nontheless! I’ve now moved to read-data-checking chunks of my repos every week (a 52th/week). Maybe I wasn’t aware of hard drives slowly dying before using restic or hard drives are just getting worse quality wise these days but I can’t remember ever having dealt with bit rot that much in the past.

7

So back then I had been using pCloud as my backend. It didn’t work well. The main issue is that any time a chunk would get partially uploaded then interrupted, pCloud would go ahead and save it instead of discarding it - and the next time Restic ran it would see the chunk, assume it was a full chunk, and continue referencing it.

I use Backblaze B2 now and haven’t had any repo issues since. Was just trying to save money - but proper object storage goes a long way, apparently. Restic has been rock solid for me on B2 and on local storage.

8

I’m too cheap for cloud services so I tunnel into a raspi in my dad’s basement and rsync my local backup there. USB hard disk and all… the rest is this story. restic has been rock solid just the crappy disks sometimes don’t take 0 and 1 too serious :joy: