Snapshots seem to be created but I always get a strange error

I’m using Restic 0.8.3, the only version I’ve ever used (I’m a new user).

After running restic for few weeks, with a crontab command like this one:

42 * * * * source /Users/gio/.restic/restic-env; /usr/local/bin/restic backup -q --exclude-file /Users/gio/.restic/exclude.txt /Users/gio; /usr/local/bin/restic forget -q --prune --keep-hourly 48 --keep-daily 30 --keep-monthly 12

I noticed that the snapshots kept increasing and prune seemed to have no effect. I checked my local mail and noticed that every backup instance was complaining that the repository (Google Cloud Storage) was locked. And it had been locked for weeks.

First question: are the snapshots created in this period to be considered valid? :thinking:

Hence I ran the unlock command and I decided to run check as well, to see if my repository was ok.

It ran for 7-8 hours and at the end, after producing few errors out, Restic commented that my repository had errors.

I decided to run rebuild-index since it seemed like a safe operation to do, according to what some other users commented in the issues on GitHub.

Now Restic again produces new snaphosts, but in the mail I always find this output:

tree 5cb11d839b361e406f30867802f34a7a26b99470b599da5ef36cd209cd4fdb86 not found in repository
github.com/restic/restic/internal/repository.(*Repository).LoadTree
        /tmp/restic-build-881482904/src/github.com/restic/restic/internal/repository/repository.go:620
github.com/restic/restic/internal/restic.FindUsedBlobs
        /tmp/restic-build-881482904/src/github.com/restic/restic/internal/restic/find.go:11
github.com/restic/restic/internal/restic.FindUsedBlobs
        /tmp/restic-build-881482904/src/github.com/restic/restic/internal/restic/find.go:31
github.com/restic/restic/internal/restic.FindUsedBlobs
        /tmp/restic-build-881482904/src/github.com/restic/restic/internal/restic/find.go:31
github.com/restic/restic/internal/restic.FindUsedBlobs
        /tmp/restic-build-881482904/src/github.com/restic/restic/internal/restic/find.go:31
github.com/restic/restic/internal/restic.FindUsedBlobs
        /tmp/restic-build-881482904/src/github.com/restic/restic/internal/restic/find.go:31
github.com/restic/restic/internal/restic.FindUsedBlobs
        /tmp/restic-build-881482904/src/github.com/restic/restic/internal/restic/find.go:31
github.com/restic/restic/internal/restic.FindUsedBlobs
        /tmp/restic-build-881482904/src/github.com/restic/restic/internal/restic/find.go:31
github.com/restic/restic/internal/restic.FindUsedBlobs
        /tmp/restic-build-881482904/src/github.com/restic/restic/internal/restic/find.go:31
github.com/restic/restic/internal/restic.FindUsedBlobs
        /tmp/restic-build-881482904/src/github.com/restic/restic/internal/restic/find.go:31
github.com/restic/restic/internal/restic.FindUsedBlobs
        /tmp/restic-build-881482904/src/github.com/restic/restic/internal/restic/find.go:31
main.pruneRepository
        src/github.com/restic/restic/cmd/restic/cmd_prune.go:191
main.runForget
        src/github.com/restic/restic/cmd/restic/cmd_forget.go:235
main.glob..func9
        src/github.com/restic/restic/cmd/restic/cmd_forget.go:24
github.com/restic/restic/vendor/github.com/spf13/cobra.(*Command).execute
        /tmp/restic-build-881482904/src/github.com/restic/restic/vendor/github.com/spf13/cobra/command.go:698
github.com/restic/restic/vendor/github.com/spf13/cobra.(*Command).ExecuteC
        /tmp/restic-build-881482904/src/github.com/restic/restic/vendor/github.com/spf13/cobra/command.go:783
github.com/restic/restic/vendor/github.com/spf13/cobra.(*Command).Execute
        /tmp/restic-build-881482904/src/github.com/restic/restic/vendor/github.com/spf13/cobra/command.go:736
main.main
        src/github.com/restic/restic/cmd/restic/main.go:69
runtime.main
        /usr/local/Cellar/go/1.10/libexec/src/runtime/proc.go:198
runtime.goexit
        /usr/local/Cellar/go/1.10/libexec/src/runtime/asm_amd64.s:2361

Second question: is this error solvable? If yes how?

Third question: how do I know for certain if my backups are valid?

Thank you

for the 3rd question did you check with

       restic -r /path/ofrep check

what is the outcome should be like this

    using temporary cache in /tmp/restic-check-cache-371990897
      create exclusive lock for repository
     load indexes
       check all packs
      check snapshots, trees and blobs
      no errors were found

I’m running it now, but after few seconds I am already seeing some not good output:

⋊> ~ restic check                                                                                                                                                                                   09:14:17
password is correct
create exclusive lock for repository
load indexes
check all packs
pack 6613ac94e9ef7412ef1a29dac115ef0b17bf9e4fcbb51820f9ed42d987edda09: not referenced in any index
pack c1afafe9e24e5f24a42e8e22c0218114110c5e5c9eb310c8f4a9b19148b79025: not referenced in any index
check snapshots, trees and blobs
error for tree 8646ae87:
  tree 8646ae87ff76f010ef51dea753c117e25c46c298de58425e3b7851312c6138e9 not found in repository
error for tree 612acfb8:
  tree 612acfb86acad8bb459cec0034f960e79e5391f89c40ed4c9d43903303c65e0c not found in repository

Do I need to create my backup from scratch?

How reliable is Restic when used on a MacBook that can frequently hop from on <–> sleep states? Can it handle backups being interrupted mid-way?

I truly like Restic however I need to understand whether I can trust it 100% for my pattern of usage.

Thanks for your help :slight_smile:

In general: yes, but in your situation: maybe not. It’s just that forget and prune did not run, because the repo was locked. I suspect that a run of backup was interrupted and left a lock file in the repo. This did not prevent other runs of backup, but forget and `prune.

The error meant that restic tried to load data from the repository, but it wasn’t there and could not be loaded. This error is not handled correctly, so restic bails out and prints a stack trace. I consider this a bug, and I’ve added an issue to track correctly handling this error condition: prune: Handle tree not found correctly · Issue #1759 · restic/restic · GitHub

So at the moment, for you, as a user, it’s not solvable. Sorry about that.

You know already for certain that there is data missing in the repo: At some point, restic uploaded some metadata in a file, and now that it would like to access it, it isn’t there any more. So it already told you (albeit in a hard-to-understand way) that the repo has an error and you cannot rely on it any more. Somehow a file vanished in the repo… I don’t have any idea how that may have happened.

This error can maybe be solved by running restic rebuild-index before running check: restic complains that there are files in the repo that it knows nothing about (they are not in the so-called “index”) and that there’s data missing, which may well be in the files that are not in the index :slight_smile:

Not very, at the moment restic does not handle being interrupted and resumed much later. This works well for local repositories, but not for anything that communicates with some cloud service. It may work, but there aren’t any guarantees.

restic is very reliable for a large number of people, I’m using it for my personal backups for the last four years. The most important thing though is regularly running restic check. If there’s anything wrong, it will tell you about it. As long as you do that (and check does not return any errors), you’re fine. If it returns errors, well, you’ve been warned :slight_smile:

2 Likes

@fd0 thank you for your very thorough reply :slight_smile:

I ran a rebuild-index again today and it indeed produced some effects:

⋊> ~ restic rebuild-index                                                                                                                                                                           13:04:50
password is correct
counting files in repo
[2:32] 100.00%  6077 / 6077 packs
finding old index files
saved new indexes as [7c426fb4 f2919f31 76caa93f]
remove 23 old index files

Now I’m running check and so far no error was produced :muscle:

I’m definitely trusting Restic reliability (that’s why I chose to use it, after doing quite some research) but of course I suspected that with cloud backups there could be some trade-offs. I’ll make sure to run checks regularly.
Somehow it would be nice if Restic was capable with dealing with such scenarios but I know that there are priorities and time is a very limited resource.

Thank you very much again for helping me out and for your work :slight_smile: