[~/.c/restic]$ restic stats --mode raw-data
repository 3a6e47e7 opened successfully, password is correct
scanning...
blob <data/9805f251> not found
I was curious how big my repo is and I ran the stats command which shows the above error. What’s the deal with that ? Is my repo broken ?
running restic check is resulting in a LOT of errors.
tree 06dbf6c9: file "com.plexapp.plugins.library.db-2019-12-01" blob 88 size could not be found
tree 06dbf6c9, blob 33334fb0: not found in index
tree 06dbf6c9, blob bc0c6046: not found in index
tree 06dbf6c9, blob 58fbe113: not found in index
tree 06dbf6c9, blob 7b2fc11b: not found in index
tree 06dbf6c9, blob a305d3a9: not found in index
tree 06dbf6c9, blob c3006a0e: not found in index
tree 06dbf6c9, blob 965c16d1: not found in index
Fatal: repository contains errors
How did this happen? What can I do ?
I guess I will report a bug:
[~]$ restic prune
repository 3a6e47e7 opened successfully, password is correct
counting files in repo
building new index for repo
[17:12] 100.00% 150137 / 150137 packs
repository contains 150137 packs (676019 blobs) with 718.767 GiB
processed 676019 blobs: 0 duplicate blobs, 0 B duplicate
load all snapshots
find data that is still in use for 23 snapshots
[0:32] 100.00% 23 / 23 snapshots
Fatal: number of used blobs is larger than number of available blobs!
Please report this error (along with the output of the ‘prune’ run) at
Sign in to GitHub · GitHub
Can anyone help ? I don’t know if my repo is broken or not
@MichaelEischer gave some advice on how one might recover from such an error in this post.
Unfortunately there is no easy fix for this error and the reason for it is (I think) still unknown. I’ve personally recovered from such an error by forgetting and pruning the affected (only a few) snapshots. This might not be possible or desirable for you.
that works fine. how can I figure out which snapshots are affected ?
If I remember correctly I searched for the affected trees by running restic cat snapshot ID on all the snapshots. I don’t think there is an easier way to do this. Maybe someone else has a better idea.
Deleting snapshots now might not even be necessary for you. I think I was able to restore files which were not affected be the missing blobs.
Thanks I did try restoring some files without issues at all. but it would be nice to know which snapshots are effected and cleaning that up 
SO I ran (fishshell)
for i in (restic snapshots | awk '{print $1}' | sed '/^\//d' | sed '/^$/d' | sed '/^ID/d' | sed '/^-/d'); restic cat snapshot $i; end
Which produced no errors at all. Is there another way ?
@foolsgold Please allow people time to answer, we all have work and family etc to manage besides trying to help our fellow restic friends
You don’t need to file an issue about this. In part because it’s most likely due to a hardware problem/malfunction/bitrot/whatever, and in part because it’s nothing new - we see these errors now and then, and do what we can to find the cause. Again, it’s likely a hardware issue or other type of cause outside of restic.
Please stand by for further responses and suggestions!
obviously if it’s an error and it persists and it can’t be fixed, then it’s not just outside restic. IMO, there should be an easy way to at least detect which snaphots are having an issue, so those can be forgotten and pruned. Seems check just told me there were errors, but nothing useful besides that.
I’m not sure what you’re trying to say. If you have a perfectly good repository on a hard drive, and I punch a hole in some bits on that hard drive, so your repository becomes corrupt, is that not a cause outside of restic? If not, please explain your rationale, because I’m not getting it.
There are ways. But having a panic attack over it isn’t going to help. Wait for people to get you some pointers on what to do.
Does check complain about pack <...> not referenced in any index or pack <...> does not exist, before all those blob not found in index errors?
Which backend do you use to store your backup repository? Did you run check some time before? Did anything unusual happen to your restic runs lately? Do / How often do you run prune?
<edit>Just noticed your Github issue: Do you use AWS S3 or just an S3-compatible service? But I guess S3 should have enough provision to avoid data corruption. </edit>
You can probably find the damaged snapshots by running restic rebuild-index to repair the damaged index and then run restic stats --mode blobs-per-file which should complain about the snapshots with missing data (the mode is important here). You can also try out whether restic backup --force ... recovers some of the missing blobs. But if its an option for you to wait a bit with repairing the repository, then I’d like to try to debug this a bit further.
I agree that the means to repair damaged repository are a bit lacking, but that’s primarily because restic goes to great lengths to avoid that such errors even occur. It is up to now unclear whether the damaged repositories are the result of hardware/operating system errors or of a tiny bug in restic. It’s rather hard to analyze as these damaged repositories only show up every now and then, but without any clear leads yet.
Thanks for responding.
there were only two types of errors:
AWS (Wasabi)
I’ve not run check before. I only ran it because the stat to see how large my repo was failed
Some of the backups had failed because there was another running. I deleted the results from crontab so I can’t check
once per week. You can see details of what I do in this post: Auto-snapshots / smart backup rentention - #8 by foolsgold
See output from this command in this gist. I don’t want to flood the thread. It fails with a SIGSEGV error and a stacktrace.
FreeBSD luffy 12.1-RELEASE-p1 FreeBSD 12.1-RELEASE-p1 GENERIC amd64
restic 0.9.6 compiled with go1.13.7 on freebsd/amd64
Thanks for your help!
So the state of your backup repository is best described as mostly fine, except for the pack file that is missing from both the repository and the index. You could give restic rebuild-index a try to see whether some blobs miraculously show up again, but judging from the check output I doubt it.
Do you have the log output of the last prune run? Does it mention any invalid files?
Please run restic backup --force ... once for all backup sets, to ensure that these actually contain all blobs for all referenced files. That will at least ensure that future backups are not affected by the missing blobs (if the lost data chunk still exist in the backup source, it will actually repair the repository).
The crash look interesting. Apparently restic found the missing blob and crashed while assembling the error message xD . The crash is fixed by https://github.com/restic/restic/pull/2668 .
For what it’s worth, I also use WASABI. Every now and then I have encountered similar errors. An index rebuild usually fixed things; worst case a rebuild followed by a forced backup. Last week I got into a situation with the “blobs larger than available” and I wasn’t able to recover. Most, if not all, big problems with a restic and WASABI have been as a result of a prune.
I have changed my configuration so that I now rclone sync my NAS rectic backup (which never has issues) to WASABI and so far that has worked without issue.
but a full monthly backup would be independent of issues, or can those persist? I’ve actually already disabled prune and forget for my backups based on what I read here… Mostly I don’t delete much stuff anyway.
@foolsgold I not sure what you mean with “full […] backup” in reference to restic. Just a normal backup ... run or do you pass some special flags? By default restic uses the latest snapshot created which has the same backup paths as a starting point to avoid rescanning every single file.
In case the index of your repository refers to missing blobs or some snapshots are missing some data blobs, then future backups might be affected unless you’ve repaired the damage with rebuild-index and backup --force ... (for every backup set!).
Do you remember whether the prune command printed something unusual such as incomplete pack file?
I don’t remember seeing that. I think it was complaining that another process already had a lock on the db (the backup didn’t finish before the cronjob for the prune/forget kicked in)