Outlook.sqlite blob size could not be found


#1

I’m running restic check --read-data --check-unused --no-cache with restic 0.9.4 compiled with go1.11.4 on darwin/amd64.

A bunch of errors simlar to the one below appears:

error for tree cd439f9c:                                                   
  tree cd439f9c: file "Outlook.sqlite.bak" blob 104 size could not be found
  tree cd439f9c: file "Outlook.sqlite.bak" blob 105 size could not be found
  tree cd439f9c: file "Outlook.sqlite.bak" blob 106 size could not be found
  tree cd439f9c: file "Outlook.sqlite.bak" blob 107 size could not be found
  tree cd439f9c: file "Outlook.sqlite.bak" blob 108 size could not be found
  tree cd439f9c: file "Outlook.sqlite.bak" blob 109 size could not be found
  tree cd439f9c: file "Outlook.sqlite.bak" blob 110 size could not be found
  tree cd439f9c: file "Outlook.sqlite.bak" blob 111 size could not be found
  tree cd439f9c: file "Outlook.sqlite.bak" blob 668 size could not be found
  tree cd439f9c: file "Outlook.sqlite.bak" blob 669 size could not be found
  tree cd439f9c: file "Outlook.sqlite.bak" blob 670 size could not be found
  tree cd439f9c: file "Outlook.sqlite.bak" blob 671 size could not be found
  tree cd439f9c: file "Outlook.sqlite.bak" blob 672 size could not be found
  tree cd439f9c: file "Outlook.sqlite.bak" blob 673 size could not be found
  tree cd439f9c: file "Outlook.sqlite.bak" blob 674 size could not be found
  tree cd439f9c: file "Outlook.sqlite.bak" blob 675 size could not be found
  tree cd439f9c, blob 1dfbf70d: not found in index
  tree cd439f9c, blob 3d0fe931: not found in index
  tree cd439f9c, blob 4d397f9c: not found in index
  tree cd439f9c, blob 71400c03: not found in index
  tree cd439f9c, blob ad00e275: not found in index
  tree cd439f9c, blob 58e59a4c: not found in index
  tree cd439f9c, blob 416b427e: not found in index
  tree cd439f9c, blob 795a43a1: not found in index
  tree cd439f9c, blob 4cd538f8: not found in index
  tree cd439f9c, blob 4efbf2c0: not found in index
  tree cd439f9c, blob 0873f640: not found in index
  tree cd439f9c, blob 05ae3410: not found in index
  tree cd439f9c, blob 4b0bcfaa: not found in index
  tree cd439f9c, blob a1b7a2b6: not found in index
  tree cd439f9c, blob 5ede0cf2: not found in index
  tree cd439f9c, blob a814e30a: not found in index  

The error appears for a lot of trees, but always indicating the same files Outlook.sqlite and Outlook.sqlite.bak.

Is it possible that restic fails to backup this file for some reason and does not notice that fail during backup?


#2

I had a similar problem in the past. You can look at this topic to see if it helps you.


#3

I tried running prune which (I think) also includes rebuild-index as suggested in that topic.

$ restic prune
enter password for repository:
repository 6d460b29 opened successfully, password is correct
counting files in repo
building new index for repo
[12:03] 100.00%  48403 / 48403 packs
repository contains 48403 packs (2741077 blobs) with 220.300 GiB
processed 2741077 blobs: 0 duplicate blobs, 0 B duplicate
load all snapshots
find data that is still in use for 41 snapshots
[4:34] 100.00%  41 / 41 snapshots
Fatal: number of used blobs is larger than number of available blobs!
Please report this error (along with the output of the 'prune' run) at
https://github.com/restic/restic/issues/new

Trying to run rebuild-index now.


#4

Yes, it does. In fact, prune ignores the index files in the repository altogether and recrawls all packs as the first phase of its operation. However, the index files don’t actually get rewritten unless prune succeeds.

It looks like one or more packs got deleted somehow, and they contained blobs that are still referenced in the repository, so you have a damaged repository. Prune is trying to be super cautious and not delete data from a repository that has structural problems, even though in this case it would probably be okay.

I would definitely suggest running rebuild-index as soon as possible. When performing new backups, the restic client uses the index files to determine what blobs are already in the repository, so that it can avoid storing them again (this is the basis for the deduplication behavior across different backup operations). This means that if a current backup tries to store one of the missing blobs, but finds that blob referenced in an index, the blob won’t be stored because the client thinks it’s already there.

The sooner you get the indexes rebuilt, the sooner that problem will go away. In fact, if all of those blobs still exist in some form on the machines being backed up, future backup operations could restore the missing blobs and effectively repair the repository.

So, after you run rebuild-index, I would let some time pass so that your source machines have the opportunity to fix at least some of the damage during their regularly-scheduled backups.


#5

Rebuilding the indexes worked.

$ restic rebuild-index
enter password for repository:
repository 6d460b29 opened successfully, password is correct
counting files in repo
[12:12] 100.00%  48403 / 48403 packs
finding old index files
saved new indexes as [7d31eb52 72d3f2d1 c317fa33 05923365 6921bd4b d9a3d2bd e96e94c4 50f11e8f 0471bc38 b68b77b1 6e6eea4a b0a5909b 091f88a5 99485d27 61c7bb84 d83d311c bb1c2a5c]                                                            
remove 17 old index files

I’ll check again tomorrow after my servers have done their nightly backups.


#6

Creating a new snapshot after rebuilding the index did not solve the issue. I guess the database file is updated so much that it creates different packs every time. I’ll just remove the snapshots as suggested in the other topic and hope that the error doesn’t come back.

Looks like I’ll have to improve my backup monitoring to catch errors earlier. :smiley:

Thanks for the help!

Edit: Looks like a single snapshot was causing all the errors. Deleted it and they’re gone now.