I just ran a restic check after pruning a repo today and received errors about a missing blob that seems to affect several snapshots.
The repo in question is stored on a local file server, however I also rclone copy that repo to b2 every night.
Since I haven’t run a sync (only a copy) to b2 in a while, I was curious if there was a way to check to see if the missing blob is still in the copy on b2 and if so what would be the steps to incorporate this back into my local copy of the repo.
Also, am I correct in my assumption that using rclone copy instead of rclone sync is generally safe with regards to not corrupting the remote copy? My working practice is that I only periodically perform syncs manually after the local repo has been pruned and passes a check.
Did check report pack files to be missing or additional pack files which are not part of the index? In either case it would be a good idea to run rebuild-index to make sure the index properly reflects the actual repository content.
You can use the find to command to search the copy on B2 for the pack files which contain a blob or tree: restic find --blob 'blob-id' --show-pack-id. That command will have to download the repository index from B2 once (to be precise: every time you switch between the B2 copy and the local copy). Once you’ve found the corresponding pack files, just copy these files into your local repository and run rebuild-index. Afterwards the repository hopefully checks out fine. Otherwise if you still have the affected files running backup --force path/to/file should also fix the repository.
I’m rather curious, how the repository got damaged. Did prune show any errors? How do you connect to the local file server? Are the damaged snapshots new or old ones, or is there no obvious pattern? Could a backup run (maybe even calling unlock) have tried to run while pruning?
So, long story short, I did a rebuild-index on the repo which was failing its check and not only did that succeed with no errors, but surprisingly everything seems okay again. restic check no longer reports errors and I was able to successfully restore the file that was previously reported to have the missing blob.
As to how I got into the error state in the first place, I don’t recall prune returning any errors however I accidentally closed the tmux pane I was running it in so I can’t say for certain. The script I run performs a check immediately following the prune and I did save that output:
using temporary cache in /tmp/restic-check-cache-638095830
repository ca8284ee opened successfully, password is correct
created new cache in /tmp/restic-check-cache-638095830
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
error for tree f6e0a265:
tree f6e0a265: file "out-T09.mp4" blob 36 size could not be found
tree f6e0a265, blob cea2c60d: not found in index
error for tree 573f65f5:
tree 573f65f5: file "out-T09.mp4" blob 36 size could not be found
tree 573f65f5, blob cea2c60d: not found in index
error for tree f30c07ca:
tree f30c07ca: file "out-T09.mp4" blob 36 size could not be found
tree f30c07ca, blob cea2c60d: not found in index
error for tree 7fff2d82:
tree 7fff2d82: file "out-T09.mp4" blob 36 size could not be found
tree 7fff2d82, blob cea2c60d: not found in index
error for tree da5993dd:
tree da5993dd: file "out-T09.mp4" blob 36 size could not be found
tree da5993dd, blob cea2c60d: not found in index
error for tree 68aac5fc:
tree 68aac5fc: file "out-T09.mp4" blob 36 size could not be found
tree 68aac5fc, blob cea2c60d: not found in index
error for tree 08e8aad8:
tree 08e8aad8: file "out-T09.mp4" blob 36 size could not be found
tree 08e8aad8, blob cea2c60d: not found in index
error for tree bb86a84c:
tree bb86a84c: file "out-T09.mp4" blob 36 size could not be found
tree bb86a84c, blob cea2c60d: not found in index
error for tree 1ecbcd73:
tree 1ecbcd73: file "out-T09.mp4" blob 36 size could not be found
tree 1ecbcd73, blob cea2c60d: not found in index
error for tree 6dab07f7:
tree 6dab07f7: file "out-T09.mp4" blob 36 size could not be found
tree 6dab07f7, blob cea2c60d: not found in index
error for tree c324a1e5:
tree c324a1e5: file "out-T09.mp4" blob 36 size could not be found
tree c324a1e5, blob cea2c60d: not found in index
error for tree ada20e23:
tree ada20e23: file "out-T09.mp4" blob 36 size could not be found
tree ada20e23, blob cea2c60d: not found in index
error for tree 8cd4c2ee:
tree 8cd4c2ee: file "out-T09.mp4" blob 36 size could not be found
tree 8cd4c2ee, blob cea2c60d: not found in index
Fatal: repository contains errors
The script connects to the repo using the rest-server which is running on a FreeBSD box with the repo itself stored in a ZFS filesystem.
Anyhow, I guess I’m happy that everything seems okay now, but I too am a little curious what went wrong (or more precisely, how rebuild-index was able to fix it).
Edit: I forgot to mention that just as an experiment I tried restoring the file mentioned in the check output before running rebuild-index and restic actually crashed. Unfortunately that output was also lost with the accidentally closed tmux pane.
Well, apparently prune just messed up rebuilding the index, but stored all rewritten packs properly on disk. The strange thing here it that after rewriting pack files, restic explicitly makes sure that everything is stored on the server and only afterwards starts to rebuild the index.