Tree not found in repository

Hi all,
I’m seeing a tree xxx not found in repository with a prune execution.
I just re-index with rebuild-index and it ends with “remove 34 old index files”, but another prune run ends with the same error: tree xxx not found in repository

The same happen after delete local cache.

The file s3::/bucket/data/9c/9c740e[…] (file name show in tree xxx not found error) doesn’t exists.
I have no error on S3 (minio) server nor on the storage subsystem

Reading the https://forum.restic.net/t/recovery-options-for-damaged-repositories I see that I could “The simplest fix is to drop snapshots that refer to missing objects”. But how I can know witch repo use the 9c740e[…] file?
Is there a restic command for find the repo id that use the 9c740e[…] file?

Thanks a lot,
Michele

Yes. First, make sure that you have updated to 0.9.5 as it contains a fix for restic find that prevents it from aborting when it can’t find a tree (which is usually why you’re running restic find in the first place).

Then, run restic find --tree $TREE_ID replacing the variable with your tree ID. The output will tell you which snapshots reference the tree.

Hi,
first of all, thanks a lot for the hit.

Just tried and seem that ALL snapshots has something that goes wrong:

~# restic find --tree 9c74[...] > restic_snapshot_error.out
grep whi restic_snapshot_error.out  | cut -d " " -f 7 | sort | uniq | wc -l
39

~# restic bucket snapshots
-------------------------------------------------------------------------------------------------
39 snapshots

This is very very strange to me
On the same S3 server (that save data on a single storage) we have other ~ 50 restic clients that save every day a lot of backups for a total of 75T used

We check buckets (with prune and reindex) about every 2 or 4 weeks on every client and we haven’t ever seen this types of problems.

Do you think that can be a restic client problem / bug?
If I remember correctly (tomorrow I’ll check better) this is a win2k3 64bit client (please not comment this old old old server version…) with a restic 64 bit

In the meanwhile I’ll create another bucket and save new data starting from that client, checking it with check and/or --read-data for see what is happening

Thanks a lot!

It’s always possible that there is a bug. (There isn’t a known bug at this time.)

It’s also possible that something else deleted the pack file containing the tree.

Consider running a backup on one of the clients. Since you have run rebuild-index, and the tree is contained in every snapshot, it is likely that the tree still exists on one of the backup clients and a backup would restore that tree to the repository.

1 Like

Yes, sure :wink:
And I hope so because it make me a lot of fear that I can’t recover any of my backups!

This sound very strange to me: why this, and only this bucket (see below)?

I forgot to say that every client has a single bucket: every client save data on its own bucket and we don’t share bucket with other clients except it’s own (for performances reasons, but this is another discussion that goes beyond this talk)

Starting from you considerations, now I’ll take another snapshot into the same bucket and on a new bucket and I’ll return to you

Thanks

1 Like

Hi,
after a reindex, prune and new backup, restic say that now I have 34 of 40 snapshots corrupted. The initial situation was 39 of 39 corrupted, so at least a good news :slight_smile:

Now we are going to create a new bucket where point new restic backups. I’ll let you know what happen after some backups+prune

Now the question: do you think that the “skip if found problems” patch (issue #2224) will be integrated also in restore command?
In this case restic could continue to restore data also if it found problems on specific blob.

Thanks a lot

I have proposed such a change and I believe that the author is open to such a patch based on his comment here: