Is a Jottacloud repo reliable? Seeking user input

akrabu · June 21, 2023, 5:19pm

So I’ve got a 5TB repo in BackBlaze B2 at the moment. Was looking into cutting my costs, but wanted to be smart about it. I previously have tried pCloud, which did not play well with Restic.

I wanted to get a feel for how reliable Restic + rclone + Jottacloud would be. Any forum users been using this combo for awhile? How often do you have repo errors? Irrecoverable errors?

$99/yr for “unlimited” (upload speeds reduced after 5TB on a sliding scale) sure beats $300/yr for B2 - but only if it’s reliable…

kapitainsky · June 21, 2023, 5:31pm

akrabu · June 21, 2023, 5:38pm

I read every article mentioning Jottacloud on this forum, but only a few over at Rclone.

At the same time, I’ve uploaded 4TB at once, over a 3 week period, with Rclone to Pcloud and never had any issue with it. I checked the hashes later, it was fine. But over time, with long sustained uploads to Pcloud using Restic + Rclone… that’s when I noticed corruption. Not that I think the issue lies with Restic - it’s just, I often use Rclone in more of a “once and done” kind of way. Whereas I have Restic uploading from multiple hosts, often simultaneously, over a variety of connections, regularly, over a long period of time - so it’s more likely to hit that error case since it’s a “constant” test, if you will.

That’s why I’m specifically asking for folks who have maintained a Restic repo with Jottacloud.

mdl · June 23, 2023, 8:58pm

I am using restic + rclone to send a snapshot daily to Jottacloud. Oldest snapshot in my repo is from january 2021. Repo size is appr 2.3Tib now.

A few times a year I get a Jottacloud error. Jottacloud support always tells me there is nothing wrong and asks me for a few actions, which I can not perform because I am not using any official Jottacloud client.

Every error so far existed for a few days maximum and does not seem to have lead to an error in my repo.

Every day, my script also does a forget, prune and check and also downloads one random file to compare it to the one my nas. They always compare the same.

fortemezzo · June 25, 2023, 12:01pm

I have used Jottacloud with Restic. I started with Kopia, but I got a lot of corruption. Then I switched to Restic and it looked better. At first anyway. Then I got corruption as well. Files that were simply not readable (even through Rclone or the UI itself). Some files became readable after a day or so, making the backup correct again. But I realized that I couldn’t trust Jottacloud, so I switched to Hetzner (with sftp) and never had an issue again.

akrabu · June 25, 2023, 4:42pm

Ah yeah, I tried Kopia with B2 even and had corruption issues. Stuck with Restic ever since.

Your report is what I was afraid of, after reading some Reddit / Rclone posts about Jottacloud. How big was your repo, out of curiosity?

akrabu · July 5, 2023, 2:50pm

Alright, I had to give it a test. My B2 repo is 3.42TiB. To avoid paying that much in egress fees, I made a new repo on a local SSD, backed up everything from scratch, then used “restic copy” to get all my old snapshots over. It transferred around 800GB, which wasn’t so bad. I then Rcloned all of this to Jottacloud.

So, with the screenshot, the top-left / purple pane is me running “rclone check” to compare the SSD with the Jottacloud copy, which should verify all the MD5 sums without needing to download the data. That checked out just fine (minus a few lock files).

The top-right / green pane is me running “restic check --read-data” on the local SSD copy, which also worked fine.

And lastly, the bottom / orange pane is me running “restic check --read-data-subset 1/10” on Jottacloud… which didn’t fare well at all.

I let it “settle” a few days, to see if that would make a difference. It did not. I receive different errors, but “check” never completes successfully.

@mdl Have you ever ran “restic check --read-data” on your repo? I tried it on two different connections and couldn’t get it to work. I’m wondering if I could Rclone the whole thing down and restore that way - but, honestly, that isn’t really a solution to me.

I swear Restic+Rclone is just never as resilient as Restic alone. Or even Rclone alone. I may test Rcloning the entire repo back down from scratch, just to see if that is truly the case.

mdl · July 5, 2023, 3:45pm

Yes, I did, but with a strange issue on my system. See for example Retry during check when pack id does not match

akrabu · July 5, 2023, 4:03pm

Ah… so this seems to specifically be a Jottacloud problem, because I’ve never seen this on B2, SFTP, or REST.

Hey @MichaelEischer / @rawtaz - is this worth a GitHub bug report about Jottacloud and Pack IDs not matching, now? @mdl and I seem to be having the same problem, per this thread. I will attempt to Rclone the whole thing back down tomorrow to rule out JUST Rclone or Jottacloud issues. Sound good?

akrabu · July 5, 2023, 4:18pm

@mdl By the way, I have experience with unreliable cloud storage - namely pCloud and SharePoint (oh yes, SharePoint is a pain with Restic).

The trick I’ve found, in a pinch, is to either copy down the config / keys folder, or make a new repo locally using --copy-chunker-params. After that, do a “restic copy || restic copy || restic copy && restic check --read-data” (with whatever switches you need, with however many attempts you want to make - the above will try three times). The difference between Restore, Dump, and Copy is that Copy is (mostly) resumable. So by chaining together multiple copy commands with ||, each time it fails it’ll resume and keep trying - and when it finally succeeds, it’ll run a check. I was able to get 3TB out of SharePoint that way, and when it finally finished, a “check --read-data” worked fine.

It’s a pain, BUT it works. Just wanted you to know, should you actually ever need to RESTORE that data, before (if ever) a fix gets implemented. But yeah, always test restoring before trusting a backup provider. Some of these cloud providers don’t report errors in uploads - they’ll just throw a fit when you try to download!

This does assume that the data is actually stored correctly in the first place. Which, the RClone “check” appears to confirm. It was NOT stored correclty on pCloud, and this method likely wouldn’t have helped.

MichaelEischer · July 5, 2023, 7:48pm

Did you try downloading some of the affected files and verify their sha256 hash? The problem is likely somewhere in the interaction between rclone and Jottacloud. I’ve never seen such problems when e.g. using rclone with a local disk.

Ewww, that sounds like some file retrievals transiently returned corrupt data, but a later try by the next copy command worked. While that workaround somewhat works, you’ll have no idea how long it will take to restore some important data with that approach.

Checking the sha256 hash of a few files that were reported as corrupt should be enough to get an idea whether Jottacloud stored the data correctly or not.

akrabu · July 5, 2023, 8:46pm

Okay. So. I ran “rclone copy” on the data/00 folder. 12.835 GiB.

Simultaneously, I was running “restic check --read-data” for comparison, and to see if I could “stress” the connection.

Here’s the output from “rclone -P --stats-one-line -v copy jc:restic-db/00 ~/tmp”:

2023-07-05 13:04:31 INFO  : 000a1d5593e451eacb066f1ff2b435073d8b64f3253a2ed4a34ddfbd3e7aa8f2: Copied (new)
2023-07-05 13:05:01 INFO  : 000d3edbaa504b31a77b0dd0e00048a63a457e51e63ce19fdf8a44d08afa0dfb: Copied (new)
2023-07-05 13:05:15 INFO  : 000597ad365c8a8699a92d2f99e207bbf16c2d38cffdd8983aa2f7ee43dad8f0: Copied (new)
2023-07-05 13:05:38 INFO  : 000d44ac276d31a338c75ce91ced44062aa76cf385ced7d1b9fe7bae52bfa4f9: Copied (new)
2023-07-05 13:05:38 INFO  : 000038e38888c5d9abef36d737fa927e5d688f4cc5c79f053b3e3f62b71483d3: Copied (new)
2023-07-05 13:06:20 INFO  : 000f9e4545a31399c6b692c710cf5bd70c82ead7cfc370a9231412b6b6b08c7f: Copied (new)
2023-07-05 13:06:26 INFO  : 0014bb9e132691e79377c78abdb047a02ebf4c907c41a3642aabb4f4087feebc: Copied (new)
2023-07-05 13:06:27 INFO  : 001542311ff2dbc851db045db69ab931d2ca58e045b88636d41102103bab159a: Copied (new)
2023-07-05 13:07:12 INFO  : 0016bfdaeacd32b0531b27368b42b7d9452e0a6da75c04ecf077b62e102da18d: Copied (new)
2023-07-05 13:07:13 INFO  : 0024fd10adfdcb40d193fd2d2dab6f694560464fbcb221c4b01fabf2ebcf0a1c: Copied (new)
2023-07-05 13:07:16 INFO  : 001afa91ad680be458097c51b8394612b04f409d44e4d142bdef6d6c9876c582: Copied (new)
2023-07-05 13:07:45 INFO  : 001dae3c59ad7dee0378209c75e996c26130864f0b107d118a714a67ba12c35c: Copied (new)
2023-07-05 13:08:02 INFO  : 0027274a98bf77365f50b3d88983f2a162899603d5a5339e0d5db4a784c9a33f: Copied (new)
2023-07-05 13:08:40 INFO  : 00261679371196b2082727fea8e45bd32f5d95ba47d2f43c9578573d303b6d20: Copied (new)
2023-07-05 13:08:50 INFO  : 002a8359d46bbc8ac71904d7acb9f8986a37e57fe45a41ae977b73eb7d529679: Copied (new)
2023-07-05 13:08:54 INFO  : 002a0a89435f82c20a555f311bb56a3564d13bf27cf4faaa9f26aaec0bbf8c92: Copied (new)
2023-07-05 13:08:55 INFO  : 0029b925d14125772f66e1021d5c2121ad787cc50306dabc529bafee90b2925c: Copied (new)
2023-07-05 13:09:37 INFO  : 0030026060ec3a827ceb7d30c86954c8a9a5cf31e130376979b8337b256c2a16: Copied (new)
2023-07-05 13:10:14 INFO  : 003d7da3ac343ffad8b6caf7510ee4d5e0e6e47eafb801a051dd7b05bd78aefa: Copied (new)
2023-07-05 13:10:25 INFO  : 003ee6fb63d163998e9fb1a1a99666ba37e6000a5d67fabd1bbfda2077a964e1: Copied (new)
2023-07-05 13:10:27 INFO  : 002c33e12cef00f2e0b0f0f651802552b6e268ec66ffe94129d652602c6c3a05: Copied (new)
2023-07-05 13:10:38 INFO  : 003d1d4344afaf3f388849e56f779353200db395bbfa1eb806e2dd7ed517dec9: Copied (new)
2023-07-05 13:11:12 INFO  : 003fadb37de69f2e77847e4c06c70c2b6b2829f011541fcff1546e2c0eaa9ec3: Copied (new)
2023-07-05 13:11:40 INFO  : 003f2972806af75e10e25b6fb1f38587a8f9f8764491d484501b978af2eb4ee2: Copied (new)
2023-07-05 13:11:49 INFO  : 004069a0996cb45071214283073cd9ba6ffb3b83ea2344424065466bd11fa0bb: Copied (new)
2023-07-05 13:11:58 INFO  : 004008de7c3317b678b0657305f4c6f0fcaee03aa887dcd65190e4e2085a0007: Copied (new)
2023-07-05 13:12:00 INFO  : 00422ca4d37d4834ad609a7cd5c74f27546b95c02f40d2f5e82177b1253608a1: Copied (new)
2023-07-05 13:12:49 INFO  : 0047be18eb9e7d49874ff3ca1117d5bc55e6c02378f36cc5853194e94cb598b6: Copied (new)
2023-07-05 13:13:11 INFO  : 00423b74c63365af515ce9a36758550584fbcafb5ed74965c124d28beabf3275: Copied (new)
2023-07-05 13:13:14 INFO  : 00452b2a4c675dad4830280a340ef25fd06d0c05f3006b93577fa667a7b5a711: Copied (new)
2023-07-05 13:14:08 INFO  : 0045dc3c8957b49a7276278273fb4d3a1d9eef9016004a25e84a50efaf61a551: Copied (new)
2023-07-05 13:14:38 INFO  : 0048d3f9f9ebdca806718306a16d8f524c790ca509fa3f9f57ec9c7ad8c82e23: Copied (new)
2023-07-05 13:14:40 INFO  : 004b0efc1765d0b00524551abd81fcd3612b87d4d8f1c2babb4180bebda30872: Copied (new)
2023-07-05 13:14:54 INFO  : 004b4b69c0a64df6fdced2de7d78222a51d147779e48c495729e6665e8218ea5: Copied (new)
2023-07-05 13:15:28 INFO  : 004ee150c95249721f24e2401a8184d47c5daf96a5df3d186905423dff96c96c: Copied (new)
2023-07-05 13:15:40 INFO  : 00502f8d8b1c222f5667a15061e61025c68c93cb0c249343f94cdd8478106f55: Copied (new)
2023-07-05 13:16:11 INFO  : 004b9c7e76c1aae0f859e2058ea1634b0113017cd9a141b37542a9be3575bbda: Copied (new)
2023-07-05 13:16:15 INFO  : 0048f5481d9b6bc3b4c856e1f0f11b40ccefb20715a5adf87381964576b123c8: Copied (new)
2023-07-05 13:16:15 INFO  : 0051bc43ae6ca8419d604a15bd6196668acd3675badd96845ec1de92982946ba: Copied (new)
2023-07-05 13:16:28 INFO  : 0052d932e369bd85b5e425739301a3bb4f3f81737d386a9de4de4a574c11f81b: Copied (new)
2023-07-05 13:17:05 INFO  : 0056c62469bf99e3451f483c1af48f3c207f452cda90fa6eb3b94400c5935076: Copied (new)
2023-07-05 13:17:16 INFO  : 0057c74b7536f1fcddfb0ab89c5a26cdcba387ef1371881262680452fe72b5a9: Copied (new)
2023-07-05 13:17:45 INFO  : 005365d583b62bbcc2ef4836fbd48cceaadf87e253be60bfd3bee30a8d936fd5: Copied (new)
2023-07-05 13:17:49 INFO  : 005391fd6ad66e11ae7421282e6729918e51255750e5c60a1746d71d900ab9b4: Copied (new)
2023-07-05 13:17:54 INFO  : 0057c7d4d7893bcb9bda519f819f9c5d1e6c533157b7244f5aeeb54603d2e529: Copied (new)
2023-07-05 13:18:04 INFO  : 005966a32b99d419e761731c9f1f716c760e3dffa282348076e85c1b4c38818d: Copied (new)
2023-07-05 13:18:42 INFO  : 006edddff3fd0ea9b52d62d9358a9c1e78fe2030279e5ebefcfbea00abc4edae: Copied (new)
2023-07-05 13:19:19 INFO  : 005c639d7a509dc8ac554639fcb4dc195a1fdbfef8dbfbf6dcb5cbf7ac778ca7: Copied (new)
2023-07-05 13:19:46 INFO  : 0073bb9fda0ca29de5c4fd1525a1dc464af661bde65aaf407ed7db8fd65d2235: Copied (new)
2023-07-05 13:20:43 INFO  : 00744cb26a5e110fc2fef622c5c0f8cf3008c218d54662908bfb3f7f7742fb4b: Copied (new)
2023-07-05 13:21:34 ERROR : 006db843e2faa213973c638b2bb5d879169652bb222e61e608876146d3b9797b: corrupted on transfer: md5 hash differ "d1a3ca693d04c85b5f6532be36710a65" vs "e56335a5c9325387a7b302ec913d6182"
2023-07-05 13:21:34 INFO  : 006db843e2faa213973c638b2bb5d879169652bb222e61e608876146d3b9797b: Removing failed copy
2023-07-05 13:21:35 ERROR : 006f9cd9ca9fe7547de5459e3d4c55bb9829d47e754a7ef5ecc69a9e2bc1ebb3: corrupted on transfer: md5 hash differ "bc060713a82e96021d6469fcc6f92dd5" vs "2c19baed14d7c174a815a777466c7942"
2023-07-05 13:21:35 INFO  : 006f9cd9ca9fe7547de5459e3d4c55bb9829d47e754a7ef5ecc69a9e2bc1ebb3: Removing failed copy
2023-07-05 13:21:58 INFO  : 00771c27cf2ef3b5ba5441bf6ba483d0d0086a62fd0a2065644be7a3b2c200b8: Copied (new)
2023-07-05 13:22:20 INFO  : 00780c641be9520b6944fc8110a6f9f0b580850d34e5205ca911ba3ac5a14d11: Copied (new)
2023-07-05 13:22:21 INFO  : 00799ca633ed480612e66ab014a2531982b5cb5ec07b7b05bed7c170dd320fb2: Copied (new)
2023-07-05 13:22:36 INFO  : 00747a85269eae352a578eba82c3c4e4769bd4747fa1a4f568b1dbc2536c2237: Copied (new)
2023-07-05 13:22:47 INFO  : 00800e590f2c9b0112c2c55890d0af25dd5dfd5b5ab97671ce596cc448cfc841: Copied (new)
2023-07-05 13:23:07 INFO  : 00843c9e56d5df62f9beec4b9998dfd2a26f51174dcb35648b9a68f2c73c3cd3: Copied (new)
2023-07-05 13:23:10 INFO  : 008467482f31735f6581738706b467daa973f5ff30547c72451f5d3205f8f2ee: Copied (new)
2023-07-05 13:23:35 INFO  : 008d9111f744d2862d1b694f690f1cc803ce681964e17ecc8a235e744364620c: Copied (new)
2023-07-05 13:23:54 INFO  : 00902ea750034ae985d2d682062a133aec6c401d6df63dfeb2c02cab356bc782: Copied (new)
2023-07-05 13:23:58 INFO  : 0091bfa5def95c7972674a57305712b78fd7db0d0d1e43a8f6f4cf62c2fb402a: Copied (new)
2023-07-05 13:24:19 INFO  : 0086714f021f5ef06cd77fee2987e19aa8163b0253c88ecea18ef99aecb92f5c: Copied (new)
2023-07-05 13:24:23 INFO  : 0093ed583afedccc8b4aa7aaba3ef0698389ecd80867452ec6594b9196de500f: Copied (new)
2023-07-05 13:24:42 INFO  : 009617e8cc68884301bef4c4c9c53e57d5d3a5081ffcc7c685d1730f04c12838: Copied (new)
2023-07-05 13:24:48 INFO  : 0096b4976b7abba51a98c6f15fab3656013194dc2669dfc462a122b73f688ad7: Copied (new)
2023-07-05 13:25:09 INFO  : 009e13f6ad3dae54433864c47fc8ed1aa36728539d759477fcf92540c6c1e8f7: Copied (new)
2023-07-05 13:25:29 INFO  : 009ea23ed704dd5546555ad17ac2687b54d3155d3c4e165e35fbed58798c2851: Copied (new)
2023-07-05 13:25:34 INFO  : 00a08cdb09d014e86a47451e0bfc77607ced4d878e71fd0e8ba411aa9e60a630: Copied (new)
2023-07-05 13:25:39 INFO  : 0099a4fd59b68e69d4845b73cf7a3a58c21e7e5bc430cab26abeed954329cec3: Copied (new)
2023-07-05 13:25:57 INFO  : 00a1ccdbdb738c02ea8121b63b00129adfbea0ed6875b363be415db7a2d47d88: Copied (new)
2023-07-05 13:26:18 INFO  : 00a60f77d953da057d928c6c42161dd942465efae15f7ea615b3ad8feedcf0a3: Copied (new)
2023-07-05 13:26:20 INFO  : 00a9f33ec0371641bec74cbc0c26723b8fb112c2905b2d21750bdfdd632e1042: Copied (new)
2023-07-05 13:26:44 INFO  : 00ad9996b78c0d214b8d604b76194455f7898a4bbaa856502e95e9a3d49ea8b6: Copied (new)
2023-07-05 13:26:56 INFO  : 00aa857ef8e220135e97b077c3a477824e2a9e153516375116e8170d6e93b056: Copied (new)
2023-07-05 13:27:03 INFO  : 00b3e95345eb9ab58883b0c8c355bd279c4541a706c3f57c71ba40a717091baa: Copied (new)
2023-07-05 13:27:07 INFO  : 00b48f9e48f2e3f5803a0c0e910fcd65e1804d65c5b8e486909eb5bd46c3eaa7: Copied (new)
2023-07-05 13:27:29 INFO  : 00b6fc92fc761e5130c8345173b8a7c4092193af2946226fc008ea879fba9cb1: Copied (new)
2023-07-05 13:27:50 INFO  : 00b7e79a098665d8b671434f132977801866f4972995552d349189d10db157db: Copied (new)
2023-07-05 13:27:55 INFO  : 00b9263a844cc0d4d8b7d0217b62001cfb8522f4dd23ccab6772ac77b2a81ed3: Copied (new)
2023-07-05 13:28:14 INFO  : 00b7acfb4837b2e356fb8c213e1d36aea7bcecc96b03a33448627080c02b4e96: Copied (new)
2023-07-05 13:28:18 INFO  : 00bede6a954e2b174988e5f97ac160a92c0cf0bc19db4a3716d9500a8682c6a9: Copied (new)
2023-07-05 13:28:37 INFO  : 00c2763acadfe8fb18ee40187a034610b152d5ea0e59704cc75a2ff0217a9bb2: Copied (new)
2023-07-05 13:28:44 INFO  : 00c8a572531b5b64b1fa29ebee0307abff5ccbfdb2e2e88614222924d3d2a292: Copied (new)
2023-07-05 13:29:04 INFO  : 00c9c83df2688dc1d9a4e689342e9ac042058db07d1023c820b2fa0d6898398d: Copied (new)
2023-07-05 13:29:23 INFO  : 00d3489935eea4b96e77e570ecd4418ba97fb83513a15561227ba52a758bdc83: Copied (new)
2023-07-05 13:29:33 INFO  : 00d39b07bf04e9de2e51b95380d2f9992f2fe3b4fb531bc944faf3853b7edd43: Copied (new)
2023-07-05 13:29:53 INFO  : 00d7ba9e4f2d6193e163164d46bcec1203b5dd3d06cf4de422a9dc35dc4aed5d: Copied (new)
2023-07-05 13:29:57 INFO  : 00c93bc6fb72ba43ace1dceb0a136137d47462ce39977dfa32b65be2906a130f: Copied (new)
2023-07-05 13:30:08 INFO  : 00dedd9fae8d37ae60377c12afb2d53e624646539ff3d47c298fcf29d7077263: Copied (new)
2023-07-05 13:30:19 INFO  : 00dfee91ba8da80afbe5fdb63c39657c2d266880395b9333d560b2caaf82c9bc: Copied (new)
2023-07-05 13:30:43 INFO  : 00e0eca140ad524d716114a22bbad9554eeb62c9ad7ea2b7978c153c75111726: Copied (new)
2023-07-05 13:30:59 INFO  : 00e37b108e521ce0bbcb406c1712a1bbd4e9d489de3cee3637df288ff1a18a53: Copied (new)
2023-07-05 13:31:02 INFO  : 00e113eeb06788d0f936337c68edc1a1652d5321c313d5aa04a3b4218b95d331: Copied (new)
2023-07-05 13:31:08 INFO  : 00e588d38a3543a2edacf47624bcefd7b48bb937b08c62e6319a80b902db8a7a: Copied (new)
2023-07-05 13:31:32 INFO  : 00e8b735ae972b0cf29d8c5a62fd0a5d341f8f45cfcfb74038146b07c99a4e93: Copied (new)
2023-07-05 13:31:47 INFO  : 00ed724054a2879e55e8e1b8dbfe232d4c9fbb285274836a9398fab76a3bb0cd: Copied (new)
2023-07-05 13:31:54 INFO  : 00f17258b9c890eec446c694ac1a81a4d1ab64a863c67214fdc9d72ca54afb1c: Copied (new)
2023-07-05 13:32:21 INFO  : 00f3fa2a24614142929d16dfb77cbcff32067e61931de1cf782f61365ede10a0: Copied (new)
2023-07-05 13:33:01 INFO  : 00f0f7e2e0006606b9d5659a33daa558a1e877dcd83d3ff03c18ec06f9d23cdf: Copied (new)
2023-07-05 13:33:27 ERROR : 00fcfbb9cf3223f88836a686fade5fe86d2e0b8ab795bd6f463a6f3c0b9f3931: corrupted on transfer: md5 hash differ "9f99fae57bfd6e2a0570267111b03812" vs "4b4ea84c61334df22d4a20bd31997707"
2023-07-05 13:33:27 INFO  : 00fcfbb9cf3223f88836a686fade5fe86d2e0b8ab795bd6f463a6f3c0b9f3931: Removing failed copy
2023-07-05 13:33:54 ERROR : 00f430ccf52d74565128b9af5fe5e2061255239c2a22086f526690579f84b169: corrupted on transfer: md5 hash differ "5be5de2e3176c9355a9b57ba8e445ab3" vs "8ab6214a3bdcc79537e6151ea2f75ebc"
2023-07-05 13:33:54 INFO  : 00f430ccf52d74565128b9af5fe5e2061255239c2a22086f526690579f84b169: Removing failed copy
2023-07-05 13:33:54 ERROR : Attempt 1/3 failed with 4 errors and: corrupted on transfer: md5 hash differ "5be5de2e3176c9355a9b57ba8e445ab3" vs "8ab6214a3bdcc79537e6151ea2f75ebc"
2023-07-05 13:34:42 INFO  : 006f9cd9ca9fe7547de5459e3d4c55bb9829d47e754a7ef5ecc69a9e2bc1ebb3: Copied (new)
2023-07-05 13:35:03 INFO  : 00fcfbb9cf3223f88836a686fade5fe86d2e0b8ab795bd6f463a6f3c0b9f3931: Copied (new)
2023-07-05 13:35:14 INFO  : 00f430ccf52d74565128b9af5fe5e2061255239c2a22086f526690579f84b169: Copied (new)
2023-07-05 13:35:29 INFO  : 006db843e2faa213973c638b2bb5d879169652bb222e61e608876146d3b9797b: Copied (new)
2023-07-05 13:35:29 ERROR : Attempt 2/3 succeeded
13.340 GiB / 13.340 GiB, 100%, 6.391 MiB/s, ETA 0s
2023/07/05 13:35:29 INFO  :    13.340 GiB / 13.340 GiB, 100%, 6.391 MiB/s, ETA 0s

Here’s the output of “restic check -v --read-data” (cancelled once Rclone stopped):

using temporary cache in /var/folders/78/z94fqn6944l3mcz4mxrwf6ycx3x4jt/T/restic-check-cache-921880212
repository 24344e18 opened (version 2, compression level max)
created new cache in /var/folders/78/z94fqn6944l3mcz4mxrwf6ycx3x4jt/T/restic-check-cache-921880212
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
rclone: 2023/07/05 13:13:30 ERROR : data/d1/d16d148d067a3603b03a3d35cc33c0838d735a5b809a1b039a3589954e292459: Didn't finish writing GET request (wrote 126873160/262259278 bytes): unexpected EOF
Load(<data/d16d148d06>, 0, 0) returned error, retrying after 551.406746ms: Copy: unexpected EOF
rclone: 2023/07/05 13:13:47 ERROR : data/d1/d16d148d067a3603b03a3d35cc33c0838d735a5b809a1b039a3589954e292459: Didn't finish writing GET request (wrote 0/262259278 bytes): unexpected EOF
Load(<data/d16d148d06>, 0, 0) returned error, retrying after 398.672823ms: Copy: unexpected EOF
Load(<data/d16d148d06>, 0, 0) operation successful after 2 retries
rclone: 2023/07/05 13:20:36 ERROR : data/fa/fae4d5fdc2a5619368b3cf4018836790a3b1bf7d4d492f63d83233c89786aafc: Didn't finish writing GET request (wrote 14679456/275136145 bytes): unexpected EOF
Load(<data/fae4d5fdc2>, 0, 0) returned error, retrying after 391.221625ms: Copy: unexpected EOF
rclone: 2023/07/05 13:20:53 ERROR : data/fa/fae4d5fdc2a5619368b3cf4018836790a3b1bf7d4d492f63d83233c89786aafc: Didn't finish writing GET request (wrote 0/275136145 bytes): unexpected EOF
Load(<data/fae4d5fdc2>, 0, 0) returned error, retrying after 693.079473ms: Copy: unexpected EOF
rclone: 2023/07/05 13:21:10 ERROR : data/fa/fae4d5fdc2a5619368b3cf4018836790a3b1bf7d4d492f63d83233c89786aafc: Didn't finish writing GET request (wrote 0/275136145 bytes): unexpected EOF
Load(<data/fae4d5fdc2>, 0, 0) returned error, retrying after 895.144195ms: Copy: unexpected EOF
Load(<data/fae4d5fdc2>, 0, 0) operation successful after 3 retries
rclone: 2023/07/05 13:32:01 ERROR : data/24/249b32b290b8f91ce55c6279e88a1061f131120b61abd781915240d4866f4347: Didn't finish writing GET request (wrote 29872944/251212684 bytes): unexpected EOF
Load(<data/249b32b290>, 0, 0) returned error, retrying after 601.292829ms: Copy: unexpected EOF
rclone: 2023/07/05 13:32:05 ERROR : data/5b/5bb9bc1403d28e875beffc5feaedc7d76b2a4c78143a63de07af26cc66872e25: Didn't finish writing GET request (wrote 31972616/243591283 bytes): unexpected EOF
Load(<data/5bb9bc1403>, 0, 0) returned error, retrying after 578.616623ms: Copy: unexpected EOF
rclone: 2023/07/05 13:32:18 ERROR : data/e5/e528a8659b2886480d6ebd282c4ec759ee755d8ef09115b57b7d7446d77d893b: Didn't finish writing GET request (wrote 0/114661449 bytes): unexpected EOF
Load(<data/e528a8659b>, 0, 0) returned error, retrying after 361.411676ms: Copy: unexpected EOF
Load(<data/e528a8659b>, 0, 0) operation successful after 1 retries
Load(<data/249b32b290>, 0, 0) operation successful after 1 retries
Load(<data/5bb9bc1403>, 0, 0) operation successful after 1 retries

And lastly here’s the output of “rhash --sha256” on the data Rclone copied - and it seems to me everything was copied down correctly:

000038e38888c5d9abef36d737fa927e5d688f4cc5c79f053b3e3f62b71483d3  000038e38888c5d9abef36d737fa927e5d688f4cc5c79f053b3e3f62b71483d3
000597ad365c8a8699a92d2f99e207bbf16c2d38cffdd8983aa2f7ee43dad8f0  000597ad365c8a8699a92d2f99e207bbf16c2d38cffdd8983aa2f7ee43dad8f0
000a1d5593e451eacb066f1ff2b435073d8b64f3253a2ed4a34ddfbd3e7aa8f2  000a1d5593e451eacb066f1ff2b435073d8b64f3253a2ed4a34ddfbd3e7aa8f2
000d3edbaa504b31a77b0dd0e00048a63a457e51e63ce19fdf8a44d08afa0dfb  000d3edbaa504b31a77b0dd0e00048a63a457e51e63ce19fdf8a44d08afa0dfb
000d44ac276d31a338c75ce91ced44062aa76cf385ced7d1b9fe7bae52bfa4f9  000d44ac276d31a338c75ce91ced44062aa76cf385ced7d1b9fe7bae52bfa4f9
000f9e4545a31399c6b692c710cf5bd70c82ead7cfc370a9231412b6b6b08c7f  000f9e4545a31399c6b692c710cf5bd70c82ead7cfc370a9231412b6b6b08c7f
0014bb9e132691e79377c78abdb047a02ebf4c907c41a3642aabb4f4087feebc  0014bb9e132691e79377c78abdb047a02ebf4c907c41a3642aabb4f4087feebc
001542311ff2dbc851db045db69ab931d2ca58e045b88636d41102103bab159a  001542311ff2dbc851db045db69ab931d2ca58e045b88636d41102103bab159a
0016bfdaeacd32b0531b27368b42b7d9452e0a6da75c04ecf077b62e102da18d  0016bfdaeacd32b0531b27368b42b7d9452e0a6da75c04ecf077b62e102da18d
001afa91ad680be458097c51b8394612b04f409d44e4d142bdef6d6c9876c582  001afa91ad680be458097c51b8394612b04f409d44e4d142bdef6d6c9876c582
001dae3c59ad7dee0378209c75e996c26130864f0b107d118a714a67ba12c35c  001dae3c59ad7dee0378209c75e996c26130864f0b107d118a714a67ba12c35c
0024fd10adfdcb40d193fd2d2dab6f694560464fbcb221c4b01fabf2ebcf0a1c  0024fd10adfdcb40d193fd2d2dab6f694560464fbcb221c4b01fabf2ebcf0a1c
00261679371196b2082727fea8e45bd32f5d95ba47d2f43c9578573d303b6d20  00261679371196b2082727fea8e45bd32f5d95ba47d2f43c9578573d303b6d20
0027274a98bf77365f50b3d88983f2a162899603d5a5339e0d5db4a784c9a33f  0027274a98bf77365f50b3d88983f2a162899603d5a5339e0d5db4a784c9a33f
0029b925d14125772f66e1021d5c2121ad787cc50306dabc529bafee90b2925c  0029b925d14125772f66e1021d5c2121ad787cc50306dabc529bafee90b2925c
002a0a89435f82c20a555f311bb56a3564d13bf27cf4faaa9f26aaec0bbf8c92  002a0a89435f82c20a555f311bb56a3564d13bf27cf4faaa9f26aaec0bbf8c92
002a8359d46bbc8ac71904d7acb9f8986a37e57fe45a41ae977b73eb7d529679  002a8359d46bbc8ac71904d7acb9f8986a37e57fe45a41ae977b73eb7d529679
002c33e12cef00f2e0b0f0f651802552b6e268ec66ffe94129d652602c6c3a05  002c33e12cef00f2e0b0f0f651802552b6e268ec66ffe94129d652602c6c3a05
0030026060ec3a827ceb7d30c86954c8a9a5cf31e130376979b8337b256c2a16  0030026060ec3a827ceb7d30c86954c8a9a5cf31e130376979b8337b256c2a16
003d1d4344afaf3f388849e56f779353200db395bbfa1eb806e2dd7ed517dec9  003d1d4344afaf3f388849e56f779353200db395bbfa1eb806e2dd7ed517dec9
003d7da3ac343ffad8b6caf7510ee4d5e0e6e47eafb801a051dd7b05bd78aefa  003d7da3ac343ffad8b6caf7510ee4d5e0e6e47eafb801a051dd7b05bd78aefa
003ee6fb63d163998e9fb1a1a99666ba37e6000a5d67fabd1bbfda2077a964e1  003ee6fb63d163998e9fb1a1a99666ba37e6000a5d67fabd1bbfda2077a964e1
003f2972806af75e10e25b6fb1f38587a8f9f8764491d484501b978af2eb4ee2  003f2972806af75e10e25b6fb1f38587a8f9f8764491d484501b978af2eb4ee2
003fadb37de69f2e77847e4c06c70c2b6b2829f011541fcff1546e2c0eaa9ec3  003fadb37de69f2e77847e4c06c70c2b6b2829f011541fcff1546e2c0eaa9ec3
004008de7c3317b678b0657305f4c6f0fcaee03aa887dcd65190e4e2085a0007  004008de7c3317b678b0657305f4c6f0fcaee03aa887dcd65190e4e2085a0007
004069a0996cb45071214283073cd9ba6ffb3b83ea2344424065466bd11fa0bb  004069a0996cb45071214283073cd9ba6ffb3b83ea2344424065466bd11fa0bb
00422ca4d37d4834ad609a7cd5c74f27546b95c02f40d2f5e82177b1253608a1  00422ca4d37d4834ad609a7cd5c74f27546b95c02f40d2f5e82177b1253608a1
00423b74c63365af515ce9a36758550584fbcafb5ed74965c124d28beabf3275  00423b74c63365af515ce9a36758550584fbcafb5ed74965c124d28beabf3275
00452b2a4c675dad4830280a340ef25fd06d0c05f3006b93577fa667a7b5a711  00452b2a4c675dad4830280a340ef25fd06d0c05f3006b93577fa667a7b5a711
0045dc3c8957b49a7276278273fb4d3a1d9eef9016004a25e84a50efaf61a551  0045dc3c8957b49a7276278273fb4d3a1d9eef9016004a25e84a50efaf61a551
0047be18eb9e7d49874ff3ca1117d5bc55e6c02378f36cc5853194e94cb598b6  0047be18eb9e7d49874ff3ca1117d5bc55e6c02378f36cc5853194e94cb598b6
0048d3f9f9ebdca806718306a16d8f524c790ca509fa3f9f57ec9c7ad8c82e23  0048d3f9f9ebdca806718306a16d8f524c790ca509fa3f9f57ec9c7ad8c82e23
0048f5481d9b6bc3b4c856e1f0f11b40ccefb20715a5adf87381964576b123c8  0048f5481d9b6bc3b4c856e1f0f11b40ccefb20715a5adf87381964576b123c8
004b0efc1765d0b00524551abd81fcd3612b87d4d8f1c2babb4180bebda30872  004b0efc1765d0b00524551abd81fcd3612b87d4d8f1c2babb4180bebda30872
004b4b69c0a64df6fdced2de7d78222a51d147779e48c495729e6665e8218ea5  004b4b69c0a64df6fdced2de7d78222a51d147779e48c495729e6665e8218ea5
004b9c7e76c1aae0f859e2058ea1634b0113017cd9a141b37542a9be3575bbda  004b9c7e76c1aae0f859e2058ea1634b0113017cd9a141b37542a9be3575bbda
004ee150c95249721f24e2401a8184d47c5daf96a5df3d186905423dff96c96c  004ee150c95249721f24e2401a8184d47c5daf96a5df3d186905423dff96c96c
00502f8d8b1c222f5667a15061e61025c68c93cb0c249343f94cdd8478106f55  00502f8d8b1c222f5667a15061e61025c68c93cb0c249343f94cdd8478106f55
0051bc43ae6ca8419d604a15bd6196668acd3675badd96845ec1de92982946ba  0051bc43ae6ca8419d604a15bd6196668acd3675badd96845ec1de92982946ba
0052d932e369bd85b5e425739301a3bb4f3f81737d386a9de4de4a574c11f81b  0052d932e369bd85b5e425739301a3bb4f3f81737d386a9de4de4a574c11f81b
005365d583b62bbcc2ef4836fbd48cceaadf87e253be60bfd3bee30a8d936fd5  005365d583b62bbcc2ef4836fbd48cceaadf87e253be60bfd3bee30a8d936fd5
005391fd6ad66e11ae7421282e6729918e51255750e5c60a1746d71d900ab9b4  005391fd6ad66e11ae7421282e6729918e51255750e5c60a1746d71d900ab9b4
0056c62469bf99e3451f483c1af48f3c207f452cda90fa6eb3b94400c5935076  0056c62469bf99e3451f483c1af48f3c207f452cda90fa6eb3b94400c5935076
0057c74b7536f1fcddfb0ab89c5a26cdcba387ef1371881262680452fe72b5a9  0057c74b7536f1fcddfb0ab89c5a26cdcba387ef1371881262680452fe72b5a9
0057c7d4d7893bcb9bda519f819f9c5d1e6c533157b7244f5aeeb54603d2e529  0057c7d4d7893bcb9bda519f819f9c5d1e6c533157b7244f5aeeb54603d2e529
005966a32b99d419e761731c9f1f716c760e3dffa282348076e85c1b4c38818d  005966a32b99d419e761731c9f1f716c760e3dffa282348076e85c1b4c38818d
005c639d7a509dc8ac554639fcb4dc195a1fdbfef8dbfbf6dcb5cbf7ac778ca7  005c639d7a509dc8ac554639fcb4dc195a1fdbfef8dbfbf6dcb5cbf7ac778ca7
006db843e2faa213973c638b2bb5d879169652bb222e61e608876146d3b9797b  006db843e2faa213973c638b2bb5d879169652bb222e61e608876146d3b9797b
006edddff3fd0ea9b52d62d9358a9c1e78fe2030279e5ebefcfbea00abc4edae  006edddff3fd0ea9b52d62d9358a9c1e78fe2030279e5ebefcfbea00abc4edae
006f9cd9ca9fe7547de5459e3d4c55bb9829d47e754a7ef5ecc69a9e2bc1ebb3  006f9cd9ca9fe7547de5459e3d4c55bb9829d47e754a7ef5ecc69a9e2bc1ebb3
0073bb9fda0ca29de5c4fd1525a1dc464af661bde65aaf407ed7db8fd65d2235  0073bb9fda0ca29de5c4fd1525a1dc464af661bde65aaf407ed7db8fd65d2235
00744cb26a5e110fc2fef622c5c0f8cf3008c218d54662908bfb3f7f7742fb4b  00744cb26a5e110fc2fef622c5c0f8cf3008c218d54662908bfb3f7f7742fb4b
00747a85269eae352a578eba82c3c4e4769bd4747fa1a4f568b1dbc2536c2237  00747a85269eae352a578eba82c3c4e4769bd4747fa1a4f568b1dbc2536c2237
00771c27cf2ef3b5ba5441bf6ba483d0d0086a62fd0a2065644be7a3b2c200b8  00771c27cf2ef3b5ba5441bf6ba483d0d0086a62fd0a2065644be7a3b2c200b8
00780c641be9520b6944fc8110a6f9f0b580850d34e5205ca911ba3ac5a14d11  00780c641be9520b6944fc8110a6f9f0b580850d34e5205ca911ba3ac5a14d11
00799ca633ed480612e66ab014a2531982b5cb5ec07b7b05bed7c170dd320fb2  00799ca633ed480612e66ab014a2531982b5cb5ec07b7b05bed7c170dd320fb2
00800e590f2c9b0112c2c55890d0af25dd5dfd5b5ab97671ce596cc448cfc841  00800e590f2c9b0112c2c55890d0af25dd5dfd5b5ab97671ce596cc448cfc841
00843c9e56d5df62f9beec4b9998dfd2a26f51174dcb35648b9a68f2c73c3cd3  00843c9e56d5df62f9beec4b9998dfd2a26f51174dcb35648b9a68f2c73c3cd3
008467482f31735f6581738706b467daa973f5ff30547c72451f5d3205f8f2ee  008467482f31735f6581738706b467daa973f5ff30547c72451f5d3205f8f2ee
0086714f021f5ef06cd77fee2987e19aa8163b0253c88ecea18ef99aecb92f5c  0086714f021f5ef06cd77fee2987e19aa8163b0253c88ecea18ef99aecb92f5c
008d9111f744d2862d1b694f690f1cc803ce681964e17ecc8a235e744364620c  008d9111f744d2862d1b694f690f1cc803ce681964e17ecc8a235e744364620c
00902ea750034ae985d2d682062a133aec6c401d6df63dfeb2c02cab356bc782  00902ea750034ae985d2d682062a133aec6c401d6df63dfeb2c02cab356bc782
0091bfa5def95c7972674a57305712b78fd7db0d0d1e43a8f6f4cf62c2fb402a  0091bfa5def95c7972674a57305712b78fd7db0d0d1e43a8f6f4cf62c2fb402a
0093ed583afedccc8b4aa7aaba3ef0698389ecd80867452ec6594b9196de500f  0093ed583afedccc8b4aa7aaba3ef0698389ecd80867452ec6594b9196de500f
009617e8cc68884301bef4c4c9c53e57d5d3a5081ffcc7c685d1730f04c12838  009617e8cc68884301bef4c4c9c53e57d5d3a5081ffcc7c685d1730f04c12838
0096b4976b7abba51a98c6f15fab3656013194dc2669dfc462a122b73f688ad7  0096b4976b7abba51a98c6f15fab3656013194dc2669dfc462a122b73f688ad7
0099a4fd59b68e69d4845b73cf7a3a58c21e7e5bc430cab26abeed954329cec3  0099a4fd59b68e69d4845b73cf7a3a58c21e7e5bc430cab26abeed954329cec3
009e13f6ad3dae54433864c47fc8ed1aa36728539d759477fcf92540c6c1e8f7  009e13f6ad3dae54433864c47fc8ed1aa36728539d759477fcf92540c6c1e8f7
009ea23ed704dd5546555ad17ac2687b54d3155d3c4e165e35fbed58798c2851  009ea23ed704dd5546555ad17ac2687b54d3155d3c4e165e35fbed58798c2851
00a08cdb09d014e86a47451e0bfc77607ced4d878e71fd0e8ba411aa9e60a630  00a08cdb09d014e86a47451e0bfc77607ced4d878e71fd0e8ba411aa9e60a630
00a1ccdbdb738c02ea8121b63b00129adfbea0ed6875b363be415db7a2d47d88  00a1ccdbdb738c02ea8121b63b00129adfbea0ed6875b363be415db7a2d47d88
00a60f77d953da057d928c6c42161dd942465efae15f7ea615b3ad8feedcf0a3  00a60f77d953da057d928c6c42161dd942465efae15f7ea615b3ad8feedcf0a3
00a9f33ec0371641bec74cbc0c26723b8fb112c2905b2d21750bdfdd632e1042  00a9f33ec0371641bec74cbc0c26723b8fb112c2905b2d21750bdfdd632e1042
00aa857ef8e220135e97b077c3a477824e2a9e153516375116e8170d6e93b056  00aa857ef8e220135e97b077c3a477824e2a9e153516375116e8170d6e93b056
00ad9996b78c0d214b8d604b76194455f7898a4bbaa856502e95e9a3d49ea8b6  00ad9996b78c0d214b8d604b76194455f7898a4bbaa856502e95e9a3d49ea8b6
00b3e95345eb9ab58883b0c8c355bd279c4541a706c3f57c71ba40a717091baa  00b3e95345eb9ab58883b0c8c355bd279c4541a706c3f57c71ba40a717091baa
00b48f9e48f2e3f5803a0c0e910fcd65e1804d65c5b8e486909eb5bd46c3eaa7  00b48f9e48f2e3f5803a0c0e910fcd65e1804d65c5b8e486909eb5bd46c3eaa7
00b6fc92fc761e5130c8345173b8a7c4092193af2946226fc008ea879fba9cb1  00b6fc92fc761e5130c8345173b8a7c4092193af2946226fc008ea879fba9cb1
00b7acfb4837b2e356fb8c213e1d36aea7bcecc96b03a33448627080c02b4e96  00b7acfb4837b2e356fb8c213e1d36aea7bcecc96b03a33448627080c02b4e96
00b7e79a098665d8b671434f132977801866f4972995552d349189d10db157db  00b7e79a098665d8b671434f132977801866f4972995552d349189d10db157db
00b9263a844cc0d4d8b7d0217b62001cfb8522f4dd23ccab6772ac77b2a81ed3  00b9263a844cc0d4d8b7d0217b62001cfb8522f4dd23ccab6772ac77b2a81ed3
00bede6a954e2b174988e5f97ac160a92c0cf0bc19db4a3716d9500a8682c6a9  00bede6a954e2b174988e5f97ac160a92c0cf0bc19db4a3716d9500a8682c6a9
00c2763acadfe8fb18ee40187a034610b152d5ea0e59704cc75a2ff0217a9bb2  00c2763acadfe8fb18ee40187a034610b152d5ea0e59704cc75a2ff0217a9bb2
00c8a572531b5b64b1fa29ebee0307abff5ccbfdb2e2e88614222924d3d2a292  00c8a572531b5b64b1fa29ebee0307abff5ccbfdb2e2e88614222924d3d2a292
00c93bc6fb72ba43ace1dceb0a136137d47462ce39977dfa32b65be2906a130f  00c93bc6fb72ba43ace1dceb0a136137d47462ce39977dfa32b65be2906a130f
00c9c83df2688dc1d9a4e689342e9ac042058db07d1023c820b2fa0d6898398d  00c9c83df2688dc1d9a4e689342e9ac042058db07d1023c820b2fa0d6898398d
00d3489935eea4b96e77e570ecd4418ba97fb83513a15561227ba52a758bdc83  00d3489935eea4b96e77e570ecd4418ba97fb83513a15561227ba52a758bdc83
00d39b07bf04e9de2e51b95380d2f9992f2fe3b4fb531bc944faf3853b7edd43  00d39b07bf04e9de2e51b95380d2f9992f2fe3b4fb531bc944faf3853b7edd43
00d7ba9e4f2d6193e163164d46bcec1203b5dd3d06cf4de422a9dc35dc4aed5d  00d7ba9e4f2d6193e163164d46bcec1203b5dd3d06cf4de422a9dc35dc4aed5d
00dedd9fae8d37ae60377c12afb2d53e624646539ff3d47c298fcf29d7077263  00dedd9fae8d37ae60377c12afb2d53e624646539ff3d47c298fcf29d7077263
00dfee91ba8da80afbe5fdb63c39657c2d266880395b9333d560b2caaf82c9bc  00dfee91ba8da80afbe5fdb63c39657c2d266880395b9333d560b2caaf82c9bc
00e0eca140ad524d716114a22bbad9554eeb62c9ad7ea2b7978c153c75111726  00e0eca140ad524d716114a22bbad9554eeb62c9ad7ea2b7978c153c75111726
00e113eeb06788d0f936337c68edc1a1652d5321c313d5aa04a3b4218b95d331  00e113eeb06788d0f936337c68edc1a1652d5321c313d5aa04a3b4218b95d331
00e37b108e521ce0bbcb406c1712a1bbd4e9d489de3cee3637df288ff1a18a53  00e37b108e521ce0bbcb406c1712a1bbd4e9d489de3cee3637df288ff1a18a53
00e588d38a3543a2edacf47624bcefd7b48bb937b08c62e6319a80b902db8a7a  00e588d38a3543a2edacf47624bcefd7b48bb937b08c62e6319a80b902db8a7a
00e8b735ae972b0cf29d8c5a62fd0a5d341f8f45cfcfb74038146b07c99a4e93  00e8b735ae972b0cf29d8c5a62fd0a5d341f8f45cfcfb74038146b07c99a4e93
00ed724054a2879e55e8e1b8dbfe232d4c9fbb285274836a9398fab76a3bb0cd  00ed724054a2879e55e8e1b8dbfe232d4c9fbb285274836a9398fab76a3bb0cd
00f0f7e2e0006606b9d5659a33daa558a1e877dcd83d3ff03c18ec06f9d23cdf  00f0f7e2e0006606b9d5659a33daa558a1e877dcd83d3ff03c18ec06f9d23cdf
00f17258b9c890eec446c694ac1a81a4d1ab64a863c67214fdc9d72ca54afb1c  00f17258b9c890eec446c694ac1a81a4d1ab64a863c67214fdc9d72ca54afb1c
00f3fa2a24614142929d16dfb77cbcff32067e61931de1cf782f61365ede10a0  00f3fa2a24614142929d16dfb77cbcff32067e61931de1cf782f61365ede10a0
00f430ccf52d74565128b9af5fe5e2061255239c2a22086f526690579f84b169  00f430ccf52d74565128b9af5fe5e2061255239c2a22086f526690579f84b169
00fcfbb9cf3223f88836a686fade5fe86d2e0b8ab795bd6f463a6f3c0b9f3931  00fcfbb9cf3223f88836a686fade5fe86d2e0b8ab795bd6f463a6f3c0b9f3931

I could try a larger dataset, or copy the whole thing down if you’d like… but it seems to me that, yes, Jottacloud has some connection errors, but Rclone alone is much more resilient than Restic+Rclone.

akrabu · July 5, 2023, 10:00pm

And yes, it is a pain - though SharePoint is what I have to deal with at work, without any other alternative other than self-hosted over the network. So I maintain “hot” local and “cold” SharePoint storages, knowing that if I need anything out of SharePoint, it might be a minute. This is where I wish Restic was a little more relentless with retries, or at least resumable with commands like Restore or Dump.

But ultimately I think it’s more the combination of Restic+Rclone. It’s been a long time, but I remember testing B2 with only Restic, and through Restic+Rclone, and I certainly had more errors with the latter than the former. But B2 is fairly reliable. This issue is much more noticeable on a less reliable backend. And yes, the backend itself being unreliable is an issue… but I feel like that should be taken more as the norm, than the exception - and Restic+Rclone could be hardened enough to be resiliently persistent, despite the unreliableness. Though I know there’s only so much you can do - like with pCloud, for instance.

akrabu · July 5, 2023, 10:42pm

Oh. And here’s the three specific pack IDs that didn’t match… which do when downloaded directly with Rclone:

MichaelEischer · July 6, 2023, 7:22pm

4 failed out of 102 total downloads is an absolutely disastrous rate. restic usually only reads parts of a pack file, such that the md5 check on rclone’s side is not possible when used together with restic.

Repository.LoadBlob currently does not retry downloads which returned corrupted pack files. I guess we can add a retry there, similar to what’s already implemented for index files and similar. Could you open an issue for that? I’ve missed that case in Redownload files with wrong hash by MichaelEischer · Pull Request #3521 · restic/restic · GitHub .

I’m not yet sure how to best handle that situation in the check command. Such a massive error rate usually indicates a completely broken storage system, which must be inspected and repaired to prevent data loss.

That’s not entirely unexpected given the rclone copy output from above. You just need a bit of luck to successfully download a pack file. (Not something I’m comfortable with in regard to backups…)

akrabu · August 5, 2023, 1:07am

So I ran “check” in 15 parts, on 15 different nights. Two failed.

Then I let it “sit” for nearly a month, untouched. Now all 15 parts check out.

I think it’s true that Jottacloud has to… settle… before you can restore. Less than ideal. Possibly workable. Not sure if I’d trust it, though…

Just a heads up should anyone come across this post, looking for answers themselves!

nicnab · August 5, 2023, 8:51am

Sorry for the late reply but when I read through the thread and then this paragraph I thought: retries are cool if you want to get your data back but totally not if you’re checking for errors. After all you do want to know about all these connection problems - or any other problems, even if they are really minor.

Right?

MichaelEischer · August 5, 2023, 9:50am

Almost. I’d find it pretty annoying if check were to complain about every minor network hickup. That would just result in too much noise such that actual problems could get ignored.

However, if LoadBlob has to retry downloading a corrupted pack file, that definitely qualifies as a problem. The only difference between check and restore would be that check ideally exits with a non-zero exit code, whereas restore could be more lenient in that regard.

nicnab · August 5, 2023, 12:40pm

D’accord! What about a “link quality” detail result like “x% of requests failed”?

akrabu · August 5, 2023, 3:53pm

I feel like that would be out of scope for Restic. It’s a backup program, after all - not a connection diagnostic program. You can generally already see the connection retries on screen. That said, it’s widely said on this forum that if you have a network problem, Restic will expose it haha

What I personally wouldn’t mind is a “–retries” switch, where you can determine just how tenacious Restic will be. This would possibly make my work-around for poor quality connections unnecessary.¹ Either that, or make restores resumable. Or ideally, both!

But back on topic… yeah, I think I’ll be sticking with B2. Jottacloud might work as some sort of “cold storage” where you realize you won’t be able to restore for a month or so after backing up, but I could also see it duplicating blobs or something like that going forward, too. My pCloud experiences have me wary of trusting a backend like this.

My work around is to do “restic copy || restic copy || restic copy || …” since Copy is resumable, and each time it fails, it will try again, until you get the full snapshot down locally. Then you can restore from the local snapshot, instead of the poor quality backend.