Hello all - I was hoping someone could point me in a useful direction.
MacOS 10.13.3
restic 0.8.2
compiled with go1.10 on darwin/amd64
backing up to locally mounted USB
I mounted my repository to check that I could restore, and to ensure that some new items were getting added.
I think I failed to unmount correctly and have caused an issue.
Now when I run a “rebuild-index” I am getting:
Load(<data/0000000000>, 591, 11701) returned error, retrying after 13.04740567s: open /Volumes/ResticLocal/BackupLocal/data/00/0000000000000000000000000000000000000000000000000000000000000000: no such file or directory
and
pack file cannot be listed 0000000000000000000000000000000000000000000000000000000000000000: open /Volumes/ResticLocal/BackupLocal/data/00/0000000000000000000000000000000000000000000000000000000000000000: no such file or directory
I don’t know the steps to correct this. I’ve run “check” and “prune” and “rebuild-index” a few times, but the problem persists.
Is there a way to determine which snapshot is referencing that “0000 etc” file and drop that snapshot?
To start with, this is how you find the index which references the pack.
for idx in $(restic list index) # for each index ID
do
# 1. cat the index
# 2. grep for a sting beginning with '0000000' (the beginning of the pack ID)
# 3. if grep finds a match, print the index ID
restic cat index $idx | grep -q '^0000000' && echo $idx
done
I’m not sure how helpful the index ID will be. Maybe you could try removing the affected index file and rebuild the index? But I would expect that “rebuild-index” would not be tripped up by a mistake in an old index file, since the purpose of the command is to assume the current index is invalid and rebuild it from an inspection of the objects in the repository.
Still, it is probably useful to know what blobs are contained within the pack.
PACK_ID="0000000000000000000000000000000000000000000000000000000000000000"
for idx in $(restic -r . list index) ; do
# here we use the 'jq' utility to parse the JSON and output only the
# contents of the object with the matching pack ID
restic -r . cat index $idx | jq '.packs[] | select( .id == "$PACK_ID" )'
done
Here’s where it gets kind of hairy. I don’t know of an efficient method for determining which files or trees are contained in the blobs that are stored in the affected pack. To figure that out, you would basically have to iterate over every snapshot in the repository and walk their entire tree looking for hits on the blob IDs in the missing pack. I would probably use a depth-first recursive search, paired with a memoization to avoid re-checking subtrees that have already been checked. When you find a hit, you can print out which snapshot/file/tree is affected by missing blob (hence the missing pack).
If you still have access to all the original data for the affected files/trees you should be able to do a “backup” to re-insert them into the repository and then run “rebuild-index”. If you don’t, you could drop the affected snapshots and then “rebuild-index” and “prune”.
This seems like another use case that might be helped by parity data within the repository.
@sitwon thanks for trying to help, but this is a different issue reported earlier today in #1641: There’s a file in the repo which does not have a valid name (the name is not a SHA-256 hash). Before, restic ignored these files, but since the last internal change this bug got introduced. It’ll be fixed with #1643.
If you like, you can have a look which files have an invalid name with the following command:
$ find /path/to/repo -type f | egrep -v '/[a-f0-9]{64}$'
The files can probably be removed, then the error will be gone.