Hi all,
I’m seeing a tree xxx not found in repository with a prune execution.
I just re-index with rebuild-index and it ends with “remove 34 old index files”, but another prune run ends with the same error: tree xxx not found in repository
The same happen after delete local cache.
The file s3::/bucket/data/9c/9c740e[…] (file name show in tree xxx not found error) doesn’t exists.
I have no error on S3 (minio) server nor on the storage subsystem
Reading the https://forum.restic.net/t/recovery-options-for-damaged-repositories I see that I could “The simplest fix is to drop snapshots that refer to missing objects”. But how I can know witch repo use the 9c740e[…] file?
Is there a restic command for find the repo id that use the 9c740e[…] file?
Yes. First, make sure that you have updated to 0.9.5 as it contains a fix for restic find that prevents it from aborting when it can’t find a tree (which is usually why you’re running restic find in the first place).
Then, run restic find --tree $TREE_ID replacing the variable with your tree ID. The output will tell you which snapshots reference the tree.
This is very very strange to me
On the same S3 server (that save data on a single storage) we have other ~ 50 restic clients that save every day a lot of backups for a total of 75T used
We check buckets (with prune and reindex) about every 2 or 4 weeks on every client and we haven’t ever seen this types of problems.
Do you think that can be a restic client problem / bug?
If I remember correctly (tomorrow I’ll check better) this is a win2k3 64bit client (please not comment this old old old server version…) with a restic 64 bit
In the meanwhile I’ll create another bucket and save new data starting from that client, checking it with check and/or --read-data for see what is happening
It’s always possible that there is a bug. (There isn’t a known bug at this time.)
It’s also possible that something else deleted the pack file containing the tree.
Consider running a backup on one of the clients. Since you have run rebuild-index, and the tree is contained in every snapshot, it is likely that the tree still exists on one of the backup clients and a backup would restore that tree to the repository.
Yes, sure
And I hope so because it make me a lot of fear that I can’t recover any of my backups!
This sound very strange to me: why this, and only this bucket (see below)?
I forgot to say that every client has a single bucket: every client save data on its own bucket and we don’t share bucket with other clients except it’s own (for performances reasons, but this is another discussion that goes beyond this talk)
Starting from you considerations, now I’ll take another snapshot into the same bucket and on a new bucket and I’ll return to you
Hi,
after a reindex, prune and new backup, restic say that now I have 34 of 40 snapshots corrupted. The initial situation was 39 of 39 corrupted, so at least a good news
Now we are going to create a new bucket where point new restic backups. I’ll let you know what happen after some backups+prune
Now the question: do you think that the “skip if found problems” patch (issue #2224) will be integrated also in restore command?
In this case restic could continue to restore data also if it found problems on specific blob.