Restic + Rclone (pCloud) connection issues

What can be done for errors like this? Rebuild-index and prune don’t seem to be cutting it… :frowning_face:

tree 786a54f4: file “2020-07-13 10.23.06.jpg” blob 2 size could not be found
tree 786a54f4: file “2020-07-13 10.23.06.jpg” blob 3 size could not be found
tree 786a54f4: file “2020-07-13 19.08.24.jpg” blob 0 size could not be found
tree 786a54f4: file “2020-07-13 19.08.24.jpg” blob 1 size could not be found
tree 786a54f4, blob dce4b0d9: not found in index
tree 786a54f4, blob f870a0d3: not found in index
tree 786a54f4, blob b8591f64: not found in index
tree 786a54f4, blob a5a9c2a3: not found in index

EDIT: Ran a rebuild-index. Running a fresh snapshot of my Mac and Drobo now. Unfortunately my grandpa’s computer that had been backing up to my repo has bit the dust. Hoping the damage isn’t in his part of the repository. Going to try the advice here after that. I’ll report back. I think I’ll be cloning the repo on pCloud from now on, and running prune on the clone followed by a check before replacing the original repo from now on…

Exactly, follow those instructions. Basically what you want to do to get the useful parts of it back into shape is running checks to find what is missing, then find the snapshots that are affected by whatever is missing, forgetting them, rebuilding and checking again.

When you check, you can check just the integrity of the repository, and I’d start with that. Once you fixed all those problems, I’d do a check --read-data to verify that all the data stored in the repository is still intact (so it hasn’t gone bad in the storage). If in the end you have a successful run of that, then your repository should be fine IMO.

I’d highly recommend using the latest master build when you do this, because it contains serious improvements/optimizations for e.g. the check command and what not. It will safe you a lot of time I think.

Best case not too many snapshots will be affected, worst case most of them will. And yeah, given what you’ve shown here I sure wouldn’t use pCloud :slight_smile:

Cool. Doing a backup --force right now. It’s got about 700GB to sort through still. Nearly all of it should already be in the repo, unless it got corrupted. After this backup is done I’ll do the check and I am indeed using the latest master build, as of today (I’ve been using one from about a month ago before this, too). Then I’ll do a little manual pruning if anything is missing, then run a check --read-data and see what that does, as you suggested.

Unfortunately I’ve already bought a lifetime account. I’m going to try continuing to use pCloud, but only prune once or twice a year - and on a cloned repository, at that. We’ll see. Hopefully over time restic + rclone webdav stability will improve. I’ve been messing with duplicacy CLI, which natively supports webdav, and it’s been working just fine with pCloud and a ~70GB repository - but I vastly prefer restic.

I’m still wondering if there isn’t a retry or low-level retry switch I should try passing to rclone via restic. Might make it more robust with the occasional network hiccup?

Ps. I should mention I’ve used pCloud for about a year now, with no issue at all - but I’ve been (justifiably) paranoid about running prune. I’ve done check --read-data about three times just to check on it occasionally with no errors at all. All this happened cause I got the idea in my head that now would be a good time to run prune since I moved and now have 100mb fiber now instead of DSL. I was, apparently, wrong. I will say it’s the same ISP, though. I don’t have this issue if I run prune from work - I just don’t like using my work connection for that. Plus we have forced updates with automatic reboots, which sometimes screws things up too.

Oh! Ha, I’m dumb. pCloud has a “rewind” feature. I’ll just restore to the day before the prune operation, and I should be set!

I am going to keep the corrupted repo up and running, as an exercise in repair, and still see what I can manage with it - but now I for sure have a viable backup :+1:

Very good. Might want to run a check --read-data after rewinding just to be sure!

1 Like

Hmm, I’m using one of the beta versions (v0.9.6-353-gfa135f72). Wonder if this isn’t a bug? A normal restic check finishes just fine, no errors minus duplicate files.

Going to try it again with v0.9.6-364-gb1b3f1ec and then stable if that doesn’t work.

This is highly interesting. It’s the same error message that @fd0 and @MichaelEischer has been debugging extensively in Restic slice bounds out of range? .

The v0.9.6-353-gfa135f72 version you tried it with is just one commit behind the latest master so that’s perfectly fine. I doubt the other versions will make a difference, and the error you are seeing now is something I am pretty sure the other guys would like to get your help investigating.

@MichaelEischer Can you sum up what patching we’d like to see (“manually compiled” text as well as the debugging output) for @akrabu to build and run a debug version for this issue?

EDIT: My bad, didn’t look closely enough - @MichaelEischer noted that this is just the progress bar crashing, it has nothing to do with the other report.

1 Like

Sure. Point me at a debug version and I’ll be happy to help troubleshoot it. :+1:

That said, it did have to run overnight before it did it, so it will be a slow process lol

Hmm, I’m at 13% with the latest master, and just got this:

Load(<data/239b887aba>, 0, 0) returned error, retrying after 720.254544ms: <data/239b887aba> does not exist

However, it does exist, both in the corrupted repo and the restored repo that I’m checking right now. MD5 checks out the same on both, as well.

Okay, so I’m going to re-run check --read-data and see if it times out at the exact same spot again. Could be pCloud randomly doesn’t serve up data, which would be unfortunate. I’m also going to rclone the entire repository to an external disk now that I’ve made room for it. Probably going to take awhile. It’s roughly 800GB - 1TB. But that way I can rule out the provider and know if this is a bug or just pCloud not being a good backend.

Top two panes is my work machine, which has a faster internet connection. Bottom is home, where I’m cloning the repo. :+1:

So the latest master AND the current stable build both give me:

pack 5c0f447c: not referenced in any index
pack c0e32d88: not referenced in any index
pack af6474b3: not referenced in any index
62 additional files were found in the repo, which likely contain duplicate data.
You can run restic prune to correct this.
check snapshots, trees and blobs
read all data
panic: runtime error: slice bounds out of range [:-5]

goroutine 3534 [running]:
main.newReadProgress.func1(0x0, 0x0, 0x0, 0x0, 0x64, 0x0, 0x125e90e801, 0x5255e00)
cmd/restic/cmd_check.go:114 +0x2f8
github.com/restic/restic/internal/restic.(*Progress).updateProgress(0xc032a791e0, 0x0, 0x0, 0x0
, 0x0, 0x64, 0x0, 0x387a00)
internal/restic/progress.go:147 +0xb4
github.com/restic/restic/internal/restic.(*Progress).Report(0xc032a791e0, 0x0, 0x0, 0x0, 0x0, 0
x1, 0x0)
internal/restic/progress.go:136 +0x15c
github.com/restic/restic/internal/checker.(*Checker).ReadPacks.func1(0xc02c680f68, 0x14a2cd5)
internal/checker/checker.go:803 +0x200
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc032a6cf00, 0xc032a6cf90)
vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66

So you were right. Not sure what to do at this point. Also not sure if it’s my repo, or a bug?

I wonder if I pass --quiet if the progress bar won’t crash?

Worth a shot. Could you also open a bug report issue on GitHub? If you don’t feel like it, let me know and I can do it, it’s just cooler if you do it.

1 Like

The crash will disappear if you pass --quiet. This bug is triggered when the status line exceeds the terminal width. Making sure that the terminal is always 40+ characters wide should also help.

I’ve opened a PR to fix this crash: https://github.com/restic/restic/pull/2899

A single Load([...]) [...] does not exist warning shouldn’t cause problems as restic retries the operation up to ten times. What worries me more is that in your screenshot the same file is listed twice.

1 Like

@MichaelEischer Does this mean there’s no sense in duplicating the issue by opening another bug report?

@rawtaz Hmm, still getting some interesting results with --quiet

pack 86831381: not referenced in any index
pack 94bf0989: not referenced in any index
pack c0e32d88: not referenced in any index
62 additional files were found in the repo, which likely contain duplicate data.
You can run restic prune to correct this.
check snapshots, trees and blobs
read all data
rclone: 2020/08/26 09:24:12 ERROR : data/1f/1f8796e28435cd6ab2092c205dd27c0ed85267b8f97990c6ee81039c11d5c56a: Didn’t finish writing GET request (wrote 620576/5138113 bytes): read tcp 10.137.61.27:60859->74.120.8.224:443: read: connection reset by peer
Load(<data/1f8796e284>, 0, 0) returned error, retrying after 720.254544ms: unexpected EOF
Pack ID does not match, want d596975c, got 79cfb256

It’s still going. Does this mean I should try to find d596975c in an earlier version and replace it if the sums don’t match??

Finally got an offline copy… and now I’m getting this. Starting to think my repo is hosed :frowning:

check snapshots, trees and blobs
error for tree 8d225dfa:
decrypting blob 8d225dfa7ce38645ef026e5b8314b6475d65c2d7f1f6e6edd9b00778b73ee794 failed: ciphertext verification failed

Hmm, even with the --quiet switch I get this now:

successfully removed locks
incomplete pack file (will be removed): 416036aa0ca7eacae9d45c1dc73f8f3e78c0b36b13e88010d77edd590547b67c
incomplete pack file (will be removed): a661505028d0772716b3fefc602b5ef63bdc692a5c155c0ab57846dde8c8f094
incomplete pack file (will be removed): af6474b3856499e64b7e3db2a19159ad482b1a0aaf7481416a5b543a45534595
incomplete pack file (will be removed): b707cc9fca1b0dc2b13fe4418fa16c3e5eac7b7f0b49f27ec8b9482548bd0f27
incomplete pack file (will be removed): b87e1edaca3363be11908cf60f1bc9dffdde363c0e402802f55c254b58ea39b6
incomplete pack file (will be removed): d596975c62d722ac6859a017839a5ec9679fd4dc5f8b5b4eb7e4d2bc502a87cf
incomplete pack file (will be removed): dbcc1edf0210249cf8be551059719ea97f992c9cbba311f3244e62565b96d877
incomplete pack file (will be removed): e7551bb95e1557fe46e96dd79b8cd5831c9488d938c84786c1874cca1931f38a
incomplete pack file (will be removed): fa169b6b79a62d6fc13e6e7ba1d59ad8d789b7d7ebdaf8d520819a2030433448
id 8d225dfa7ce38645ef026e5b8314b6475d65c2d7f1f6e6edd9b00778b73ee794 not found in repository
github.com/restic/restic/internal/repository.(*Repository).LoadBlob
github.com/restic/restic/internal/repository/repository.go:152
github.com/restic/restic/internal/repository.(*Repository).LoadTree
github.com/restic/restic/internal/repository/repository.go:723
github.com/restic/restic/internal/restic.FindUsedBlobs
github.com/restic/restic/internal/restic/find.go:19
main.getUsedBlobs
github.com/restic/restic/cmd/restic/cmd_prune.go:276
main.pruneRepository
github.com/restic/restic/cmd/restic/cmd_prune.go:158
main.runPrune
github.com/restic/restic/cmd/restic/cmd_prune.go:62
main.glob…func18
github.com/restic/restic/cmd/restic/cmd_prune.go:27
github.com/spf13/cobra.(*Command).execute
github.com/spf13/cobra@v0.0.3/command.go:762
github.com/spf13/cobra.(*Command).ExecuteC
github.com/spf13/cobra@v0.0.3/command.go:852
github.com/spf13/cobra.(*Command).Execute
github.com/spf13/cobra@v0.0.3/command.go:800
main.main
github.com/restic/restic/cmd/restic/main.go:86
runtime.main
runtime/proc.go:203
runtime.goexit
runtime/asm_amd64.s:1373

Okay. I rclone’d over my local repo, and started over. This time instead of rebuild-index and prune both, I just ran rebuild-index. Then check. Now I essentially have two errors:

repository 24344e18 opened successfully, password is correct
created new cache in /var/folders/lj/z4b97q0n1wb64qrrb0rnnrbw0000gn/T/restic-check-cache-546465070
create exclusive lock for repository
load indexes
check all packs
pack d596975c: not referenced in any index
pack 416036aa: not referenced in any index
pack dbcc1edf: not referenced in any index
pack af6474b3: not referenced in any index
pack e7551bb9: not referenced in any index
pack a6615050: not referenced in any index
pack b707cc9f: not referenced in any index
pack b87e1eda: not referenced in any index
pack fa169b6b: not referenced in any index
9 additional files were found in the repo, which likely contain duplicate data.
You can run restic prune to correct this.
check snapshots, trees and blobs
error for tree 8d225dfa:
id 8d225dfa7ce38645ef026e5b8314b6475d65c2d7f1f6e6edd9b00778b73ee794 not found in repository
error for tree 786a54f4:
tree 786a54f4: file “2020-07-13 10.23.06.jpg” blob 2 size could not be found
tree 786a54f4: file “2020-07-13 10.23.06.jpg” blob 3 size could not be found
tree 786a54f4: file “2020-07-13 19.08.24.jpg” blob 0 size could not be found
tree 786a54f4: file “2020-07-13 19.08.24.jpg” blob 1 size could not be found
tree 786a54f4, blob dce4b0d9: not found in index
tree 786a54f4, blob f870a0d3: not found in index
tree 786a54f4, blob b8591f64: not found in index
tree 786a54f4, blob a5a9c2a3: not found in index
read all data
[9:16:04] 100.00% 223668 / 223668 packs
Fatal: repository contains errors

akrabu@akrabu-macbook-air ~ % restic -r /Volumes/Archive/Backup/restic-repo/ find --tree
8d225dfa 786a54f4
repository 24344e18 opened successfully, password is correct
Found tree 786a54f469ff29ac043b21b9116f12867cae175e7db84097a219f2597488a5db
… path /C/Users/Herb/Dropbox/Camera Uploads
… in snapshot 4ae87449 (2020-07-27 10:41:49)
Unable to load tree 8d225dfa7ce38645ef026e5b8314b6475d65c2d7f1f6e6edd9b00778b73ee794
… which belongs to snapshot 6afeb332bf6b9c880cf71eab20791b5812750bc46bd06b66a4c79e57f5
5395f4.

restic stats -r /Volumes/Archive/Backup/restic-repo/
akrabu@akrabu-macbook-air ~ % restic stats -r /Volumes/Archive/Backup/restic-repo/
repository 24344e18 opened successfully, password is correct
scanning…
walking tree 8d225dfa7ce38645ef026e5b8314b6475d65c2d7f1f6e6edd9b00778b73ee794: id 8d225df
a7ce38645ef026e5b8314b6475d65c2d7f1f6e6edd9b00778b73ee794 not found in repository

Okay I forgot 6afeb332 and 4ae87449. Let’s see what check --read-data does this time.

Hooray! Working repository. No errors. I rclone’d that up to pCloud. Going to try a local prune and another check. If that works out I think I’m set.

I’m still very curious about the “ciphertext verification failed” earlier, but all that is now gone.

The PR is already merged by now, so no there’s no need to open an issue now.

These messages contradict each other. Did you run both commands on the same repository copy? Did you run rebuild-index in-between?

The switch only helped with the specific crash in the check command.

The first time, I rclone’d my repo from pCloud to a local drive, ran rebuild-index, then prune, then check --read-data - and got the ciphertext error.

I then re-rclone’d over my local copy to reset it from pCloud, and this time tried just rebuild-index, then check. Afterwards I was left with only two bad snapshots. I removed those two snapshots, ran a check --read-data, and was finally left with a working repo - which I then rclone’d back into pCloud.

I think I’m going to keep this local copy, and only run prune jobs locally after rcloning down from the cloud, then only uploading back once check --read-data gives me the all clear. Not sure if it’s my connection or pCloud or both, but it’s just too unstable for prune jobs with restic+rclone. Never had this problem with b2 - so it’s either pCloud (likely) or the way restic+rclone handles things a little differently than just restic alone.

I will say, for long copy jobs, rclone often throws a lot of errors on both b2 and pCloud on my connection. It seems to handle the small sync changes well enough, though. So I can’t completely rule out CenturyLink as the culprit (with restic alone handling the poor connection better than restic+rclone).