Backblaze B2 consistency issues?

Getting Help
1

Since adopting restic a month ago, I’ve had two repositories fail checks (out of less than 5). I’ve been able to resolve these by running my backup and prune scripts again, but it leaves me wondering if B2 maybe isn’t an ideal provider for restic backups.

Here’s an example of the output of my check command:

Create exclusive lock for repository
Load indexes
Check all packs
pack b4242630b1b4234b1369f1acd4fa51d2681b68c7055b629093fa6d9cbc682dec: not referenced in any index
Check snapshots, trees and blobs
error for tree 9c51a88c:
  tree 9c51a88c, blob 6127c957: not found in index
  tree 9c51a88c, blob b83809b7: not found in index
... more of the above (error for tree)...
Fatal: repository contains errors

I’m currently on version 0.7.3 for the error I pasted above. The other client is on 0.7.2, and no hosts have had their client upgraded (nor do any hosts share a repository, though they are all in hte same B2 bucket.) I also verified that I don’t have Backblaze deleting files via lifecycle rules in that bucket.

I LOVE that restic is giving me insight into backups not being safe/trusted, but two failures in a month has me spooked. Any advice would be appreciated.

2

Hmm, the error is odd and unexpected, I didn’t get any reports from other people using B2. And it sounds suspiciously like the bug restic 0.7.0 had: https://restic.github.io/blog/2017-07-25/please-upgrade-to-0.7.1

Did you use restic 0.7.0 with this repo?

For the error you pasted: try running rebuild-index and check again, I suspect that the blobs restic is missing is in the file that’s not contained in the index…

3

I have been seeing similar issues on my new restic backups on B2. Also this sort of error;

3 snapshots have been removed, running prune
counting files in repo
building new index for repo
pack file cannot be listed 5757c0da: Stat: unexpected end of JSON input
pack file cannot be listed f0f24686: Stat: unexpected end of JSON input
[41:56] 100.00%  5608 / 5608 packs

repository contains 5606 packs (67606 blobs) with 27.085 GiB bytes
processed 67606 blobs: 0 duplicate blobs, 0B duplicate
load all snapshots
find data that is still in use for 9 snapshots
unexpected end of JSON input
4

As described in https://github.com/restic/restic/issues/1377, your issue is different as it is caused by a bug in the b2 backend (or rather the library we’re using to access b2), the error Stat: unexpected end of JSON input hints at that. It’s resolved in the master branch.

1 Like
5

FYI - I have been seeing the same not found in index errors as well on B2. Will run a new prune using your latest beta and report back if there are any issues. Thanks for the very prompt resolution :).

6

I don’t believe I’ve ever had 0.7.0 installed on these hosts. It looks like 0.7.1 was released in July and I adopted Restic in September. I do currently have some hosts on 0.7.2 and 0.7.3. I’ve scheduled a check on each of these daily to see if the problem comes up again. If so, I’ll try rebuilding the index and re-post if the problem is not resolved.