Damaged repository

Getting Help
1

One of my restic repositories has started showing errors on a check operation.

I have run a rebuild-index which has completed without error, but a subsequent check gives a swathe of errors, which I am interpreting as the data for two files in multiple snapshots is missing.

How can I best resolve this, please?

This is with restic 0.94 (error I think would have appeared with 0.93 as I have just updated).

$ restic check ....
using temporary cache in /tmp/restic-check-cache-131262506
created new cache in /tmp/restic-check-cache-131262506
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
error for tree 426a09fe:
  tree 426a09fe: file "dbd-sqlite" blob 0 size could not be found
  tree 426a09fe, blob 33490fa3: not found in index
error for tree 3f8faebf:
  tree 3f8faebf: file "dbd-sqlite" blob 0 size could not be found
  tree 3f8faebf, blob 33490fa3: not found in index
error for tree b9eb9614:
  tree b9eb9614: file "dbd-sqlite" blob 0 size could not be found
  tree b9eb9614: file "distzilla" blob 1 size could not be found
  tree b9eb9614, blob 33490fa3: not found in index
  tree b9eb9614, blob 1c14b418: not found in index
error for tree b8b7650a:
  tree b8b7650a: file "dbd-sqlite" blob 0 size could not be found
  tree b8b7650a, blob 33490fa3: not found in index
error for tree 619afcb0:
  tree 619afcb0: file "dbd-sqlite" blob 0 size could not be found
  tree 619afcb0, blob 33490fa3: not found in index
error for tree db0446d6:
  tree db0446d6: file "dbd-sqlite" blob 0 size could not be found
  tree db0446d6, blob 33490fa3: not found in index
error for tree 55cc722f:
  tree 55cc722f: file "dbd-sqlite" blob 0 size could not be found
  tree 55cc722f, blob 33490fa3: not found in index
error for tree 82308173:
  tree 82308173: file "dbd-sqlite" blob 0 size could not be found
  tree 82308173: file "distzilla" blob 1 size could not be found
  tree 82308173, blob 33490fa3: not found in index
  tree 82308173, blob 1c14b418: not found in index
error for tree 2a345c7e:
  tree 2a345c7e: file "dbd-sqlite" blob 0 size could not be found
  tree 2a345c7e, blob 33490fa3: not found in index
error for tree 16ae8017:
  tree 16ae8017: file "dbd-sqlite" blob 0 size could not be found
  tree 16ae8017, blob 33490fa3: not found in index
error for tree eec45ffb:
  tree eec45ffb: file "dbd-sqlite" blob 0 size could not be found
  tree eec45ffb: file "distzilla" blob 1 size could not be found
  tree eec45ffb, blob 33490fa3: not found in index
  tree eec45ffb, blob 1c14b418: not found in index
error for tree 175afd34:
  tree 175afd34: file "dbd-sqlite" blob 0 size could not be found
  tree 175afd34, blob 33490fa3: not found in index
error for tree d6501418:
  tree d6501418: file "dbd-sqlite" blob 0 size could not be found
  tree d6501418, blob 33490fa3: not found in index
error for tree 06073636:
  tree 06073636: file "dbd-sqlite" blob 0 size could not be found
  tree 06073636, blob 33490fa3: not found in index
error for tree 4b5e9993:
  tree 4b5e9993: file "dbd-sqlite" blob 0 size could not be found
  tree 4b5e9993, blob 33490fa3: not found in index
error for tree c2822519:
  tree c2822519: file "dbd-sqlite" blob 0 size could not be found
  tree c2822519, blob 33490fa3: not found in index
error for tree a5a6c1c7:
  tree a5a6c1c7: file "dbd-sqlite" blob 0 size could not be found
  tree a5a6c1c7: file "distzilla" blob 1 size could not be found
  tree a5a6c1c7, blob 33490fa3: not found in index
  tree a5a6c1c7, blob 1c14b418: not found in index
error for tree 831d4579:
  tree 831d4579: file "dbd-sqlite" blob 0 size could not be found
  tree 831d4579, blob 33490fa3: not found in index
error for tree 40beeca2:
  tree 40beeca2: file "dbd-sqlite" blob 0 size could not be found
  tree 40beeca2: file "distzilla" blob 1 size could not be found
  tree 40beeca2, blob 33490fa3: not found in index
  tree 40beeca2, blob 1c14b418: not found in index
error for tree b10f2e68:
  tree b10f2e68: file "dbd-sqlite" blob 0 size could not be found
  tree b10f2e68: file "distzilla" blob 1 size could not be found
  tree b10f2e68, blob 33490fa3: not found in index
  tree b10f2e68, blob 1c14b418: not found in index
Fatal: repository contains errors
2

Before you worry too much about figuring out what snapshots might be affected, consider running another backup of the system where these files (dbd-sqlite and distzilla) live. Since you have run rebuild-index, the indexes should not contain these blob IDs anymore. Subsequent backups with the same content of these files will therefore re-add them to the repository and fix the damage.

3

I’ve tried that, with no effect. BTW both those files are likely to be part of the dependancy modules pulled in within a perl dev project - impossible to identify which one of several projects from just those names, although all the matches I see on my current system are directories rather than files.

4

In that case, you will likely need to remove whatever snapshots refer to these files. You should be able to get a list by searching for the missing blob IDs:

restic find --blob 33490fa3 1c14b418

You should get a report about what snapshots refer to these blobs. You can then forget those snapshots to fix the problem.

5

So it appears that these two files are actually mail folders (from Postbox app), for a pair of mailing lists I unsubbed from a while back… so those files have been static for a good while.

The bad point about this is that this pair of files have been unchanged in every snapshot in the repository! So forgetting all affected snapshots would wipe out just about a year of backups.

I was wondering if the Mac full-disk-access issue was blocking the mail cache data from being picked up, but it definitely looks to me as though I have it correctly configured so that these files can be picked up, and I am not getting any of the access denied error messages.

Is there a way I can snip those two files out of the snapshots, leaving the snapshots otherwise unchanged?

Long term I am probably going to exclude that directory tree from backups - they are basically caches of remote data, which can be fairly quickly rebuilt. The mail content is backed up by other means (gmvault to a directory on a storage server that is then snapshotted by restic), so this copy is very much a relatively fast changing temporary cache, which is a bad thing to bloat my main backups with.

6

If the files are still present and unchanged, trying to back them up to the repository should restore them.

@fd0 This might call for a new feature, where specific objects can be flagged as “I know they’re missing, and they’re not critical so please ignore their absence (unless a future backup puts them back).”

1 Like
7

It appears those files were getting excluded when using my normal backup command line.

Explicitly backing them up with no exclude lines etc, has mostly fixed it - one of them has changed, and so there are half a dozen old snapshots that has that one in place.

I’ll have to live with the substantially smaller error set for the foreseeable future.