Restic check reports errors - troubleshooting guide does not solve

T-6 · September 24, 2023, 2:05pm

When doing a regular “restic check --read-data-subset” one missing StreamPack was reported.
I tried the steps from here https://restic.readthedocs.io/en/stable/077_troubleshooting.html
After step five I do again a “restic check --read-data” and get a couple of this:

Load(<data/d7d3766a3c>, 4620876, 0) returned error, retrying after 641.882316ms: <data/d7d3766a3c> does not exist
Load(<data/d7d3766a3c>, 4620876, 0) returned error, retrying after 639.386189ms: <data/d7d3766a3c> does not exist
Load(<data/d7d3766a3c>, 4620876, 0) returned error, retrying after 884.007383ms: <data/d7d3766a3c> does not exist
Load(<data/d7d3766a3c>, 4620876, 0) returned error, retrying after 925.624902ms: <data/d7d3766a3c> does not exist
Load(<data/d7d3766a3c>, 4620876, 0) returned error, retrying after 3.067747785s: <data/d7d3766a3c> does not exist
Load(<data/d7d3766a3c>, 4620876, 0) returned error, retrying after 3.236876867s: <data/d7d3766a3c> does not exist
Load(<data/d7d3766a3c>, 4620876, 0) returned error, retrying after 5.586993284s: <data/d7d3766a3c> does not exist
Load(<data/d7d3766a3c>, 4620876, 0) returned error, retrying after 5.319167079s: <data/d7d3766a3c> does not exist
Load(<data/d7d3766a3c>, 4620876, 0) returned error, retrying after 14.99696487s: <data/d7d3766a3c> does not exist
Load(<data/d7d3766a3c>, 4620876, 0) returned error, retrying after 19.861739285s: <data/d7d3766a3c> does not exist
pack d7d3766a3ca5ec9db8ee5349071751be05962e1ddb19fe3f1ecdce50a25d238a failed to download: StreamPack: <data/d7d3766a3c> does not exist
Load(<data/47b0b9dc5f>, 4583344, 0) returned error, retrying after 601.45981ms: <data/47b0b9dc5f> does not exist
Load(<data/47b0b9dc5f>, 4583344, 0) returned error, retrying after 807.252381ms: <data/47b0b9dc5f> does not exist
Load(<data/47b0b9dc5f>, 4583344, 0) returned error, retrying after 995.912297ms: <data/47b0b9dc5f> does not exist
Load(<data/47b0b9dc5f>, 4583344, 0) returned error, retrying after 1.596425081s: <data/47b0b9dc5f> does not exist
Load(<data/47b0b9dc5f>, 4583344, 0) returned error, retrying after 2.621852804s: <data/47b0b9dc5f> does not exist
Load(<data/47b0b9dc5f>, 4583344, 0) returned error, retrying after 4.623825112s: <data/47b0b9dc5f> does not exist
Load(<data/47b0b9dc5f>, 4583344, 0) returned error, retrying after 3.163012738s: <data/47b0b9dc5f> does not exist
Load(<data/47b0b9dc5f>, 4583344, 0) returned error, retrying after 6.473451449s: <data/47b0b9dc5f> does not exist
Load(<data/47b0b9dc5f>, 4583344, 0) returned error, retrying after 11.238481733s: <data/47b0b9dc5f> does not exist
Load(<data/47b0b9dc5f>, 4583344, 0) returned error, retrying after 9.830829705s: <data/47b0b9dc5f> does not exist
pack 47b0b9dc5fbddae98c25131290b0b99b5845f60573807d3f34a505cacc3465f8 failed to download: StreamPack: <data/47b0b9dc5f> does not exist
Load(<data/37f19cc057>, 5281707, 0) returned error, retrying after 645.853538ms: <data/37f19cc057> does not exist
Load(<data/37f19cc057>, 5281707, 0) returned error, retrying after 396.765825ms: <data/37f19cc057> does not exist
Load(<data/37f19cc057>, 5281707, 0) returned error, retrying after 1.103083509s: <data/37f19cc057> does not exist
Load(<data/37f19cc057>, 5281707, 0) returned error, retrying after 1.452368083s: <data/37f19cc057> does not exist
Load(<data/37f19cc057>, 5281707, 0) returned error, retrying after 2.686017589s: <data/37f19cc057> does not exist
Load(<data/37f19cc057>, 5281707, 0) returned error, retrying after 3.293190819s: <data/37f19cc057> does not exist
Load(<data/37f19cc057>, 5281707, 0) returned error, retrying after 5.8244428s: <data/37f19cc057> does not exist
Load(<data/37f19cc057>, 5281707, 0) returned error, retrying after 10.220009289s: <data/37f19cc057> does not exist
Load(<data/37f19cc057>, 5281707, 0) returned error, retrying after 11.872387083s: <data/37f19cc057> does not exist
Load(<data/37f19cc057>, 5281707, 0) returned error, retrying after 27.978956651s: <data/37f19cc057> does not exist
pack 37f19cc0576d5fb2ae1709c4e1e5afd3bd0560d4d82063120297d4f4f925252c failed to download: StreamPack: <data/37f19cc057> does not exist
Load(<data/5829fa5295>, 5349512, 0) returned error, retrying after 599.907606ms: <data/5829fa5295> does not exist
Load(<data/5829fa5295>, 5349512, 0) returned error, retrying after 459.423489ms: <data/5829fa5295> does not exist
Load(<data/5829fa5295>, 5349512, 0) returned error, retrying after 1.37996285s: <data/5829fa5295> does not exist
Load(<data/5829fa5295>, 5349512, 0) returned error, retrying after 2.493727379s: <data/5829fa5295> does not exist
Load(<data/5829fa5295>, 5349512, 0) returned error, retrying after 1.324304777s: <data/5829fa5295> does not exist
Load(<data/5829fa5295>, 5349512, 0) returned error, retrying after 4.420319333s: <data/5829fa5295> does not exist
Load(<data/5829fa5295>, 5349512, 0) returned error, retrying after 3.836243304s: <data/5829fa5295> does not exist
Load(<data/5829fa5295>, 5349512, 0) returned error, retrying after 11.891696636s: <data/5829fa5295> does not exist
Load(<data/5829fa5295>, 5349512, 0) returned error, retrying after 11.703673738s: <data/5829fa5295> does not exist
Load(<data/5829fa5295>, 5349512, 0) returned error, retrying after 22.017452869s: <data/5829fa5295> does not exist
pack 5829fa529592838aab3ad4b03bcb3eb25d8d3290df18134bf5b7e6f8aa59d37e failed to download: StreamPack: <data/5829fa5295> does not exist

I searched for the files and they are present. When I compare SHA265 the sum does not match the filename.

Backup is done from NAS (Debian 11) to pCloud using restic/rclone. Version of restic:

16:06:50 root@server:/usr/local/bin# restic version
restic 0.16.0 compiled with go1.20.6 on linux/amd64

As the troubleshooting guide suggests, I am seeking for help here
Thanks

KamikazeePL · September 25, 2023, 7:40pm

Maybe permissions problem?

MichaelEischer · September 26, 2023, 8:05pm

This might be a problem with pCloud as reported in a few other forum threads. Please check whether there are other file versions of the files that have been reported as missing.

Are those non-existent pack files the only problem reported by check?

T-6 · September 29, 2023, 1:38pm

Actually pCloud has been quite stable for me after moving to the European server.

I checked the files in pCloud. There are no previous versions. The files are all older than 2 years and versioning on pCloud is only for changes in the last 15/30 days:
pCloud Versioning

Here is the complete error message of one check:

using temporary cache in /tmp
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[1:52] 100.00%  45 / 45 snapshots

read all data
Load(<data/c172d6f41f>, 4295748, 0) returned error, retrying after 294.076872ms: <data/c172d6f41f> does not exist
Load(<data/c172d6f41f>, 4295748, 0) returned error, retrying after 526.254841ms: <data/c172d6f41f> does not exist
Load(<data/c172d6f41f>, 4295748, 0) returned error, retrying after 936.638653ms: <data/c172d6f41f> does not exist
Load(<data/c172d6f41f>, 4295748, 0) returned error, retrying after 1.626677595s: <data/c172d6f41f> does not exist
Load(<data/c172d6f41f>, 4295748, 0) returned error, retrying after 2.215909107s: <data/c172d6f41f> does not exist
Load(<data/c172d6f41f>, 4295748, 0) returned error, retrying after 3.43997022s: <data/c172d6f41f> does not exist
Load(<data/c172d6f41f>, 4295748, 0) returned error, retrying after 3.963866983s: <data/c172d6f41f> does not exist
Load(<data/c172d6f41f>, 4295748, 0) returned error, retrying after 11.738840011s: <data/c172d6f41f> does not exist
Load(<data/c172d6f41f>, 4295748, 0) returned error, retrying after 6.498630431s: <data/c172d6f41f> does not exist
Load(<data/c172d6f41f>, 4295748, 0) returned error, retrying after 19.026029467s: <data/c172d6f41f> does not exist
pack c172d6f41f0fa837bbda2e3ad8525ed28e6c323e6b55c97320c49d8c92bc6bb4 failed to download: StreamPack: <data/c172d6f41f> does not exist
Load(<data/43776ee989>, 4477573, 0) returned error, retrying after 665.045653ms: <data/43776ee989> does not exist
Load(<data/43776ee989>, 4477573, 0) operation successful after 1 retries
Load(<data/93675b67cc>, 4578500, 0) returned error, retrying after 597.082404ms: <data/93675b67cc> does not exist
Load(<data/93675b67cc>, 4578500, 0) operation successful after 1 retries
Load(<data/f327dfc324>, 4620876, 0) returned error, retrying after 403.660267ms: <data/f327dfc324> does not exist
Load(<data/f327dfc324>, 4620876, 0) operation successful after 1 retries
Load(<data/83b765f942>, 5967819, 0) returned error, retrying after 694.108931ms: <data/83b765f942> does not exist
Load(<data/83b765f942>, 5967819, 0) operation successful after 1 retries
Load(<data/634bdc5341>, 6283015, 0) returned error, retrying after 633.160879ms: <data/634bdc5341> does not exist
Load(<data/634bdc5341>, 6283015, 0) operation successful after 1 retries
Load(<data/2380590ed6>, 5310569, 0) returned error, retrying after 628.995526ms: <data/2380590ed6> does not exist
Load(<data/2380590ed6>, 5310569, 0) operation successful after 1 retries
Load(<data/43e3abad4f>, 6134553, 0) returned error, retrying after 421.877394ms: <data/43e3abad4f> does not exist
Load(<data/43e3abad4f>, 6134553, 0) operation successful after 1 retries
Load(<data/b413a6a3ed>, 4456831, 0) returned error, retrying after 744.910675ms: <data/b413a6a3ed> does not exist
Load(<data/b413a6a3ed>, 4456831, 0) returned error, retrying after 1.094076833s: <data/b413a6a3ed> does not exist
Load(<data/b413a6a3ed>, 4456831, 0) returned error, retrying after 924.270768ms: <data/b413a6a3ed> does not exist
Load(<data/b413a6a3ed>, 4456831, 0) returned error, retrying after 1.638013528s: <data/b413a6a3ed> does not exist
Load(<data/b413a6a3ed>, 4456831, 0) returned error, retrying after 2.556343112s: <data/b413a6a3ed> does not exist
Load(<data/b413a6a3ed>, 4456831, 0) returned error, retrying after 3.944394551s: <data/b413a6a3ed> does not exist
Load(<data/b413a6a3ed>, 4456831, 0) returned error, retrying after 8.042174848s: <data/b413a6a3ed> does not exist
Load(<data/b413a6a3ed>, 4456831, 0) returned error, retrying after 8.944095478s: <data/b413a6a3ed> does not exist
Load(<data/b413a6a3ed>, 4456831, 0) returned error, retrying after 15.231117299s: <data/b413a6a3ed> does not exist
Load(<data/b413a6a3ed>, 4456831, 0) returned error, retrying after 20.544602602s: <data/b413a6a3ed> does not exist
pack b413a6a3ed36350eae0ac70f0ad6f2ca8fa93e1b06886d707fb9dbc35615a35a failed to download: StreamPack: <data/b413a6a3ed> does not exist
Load(<data/564e598c64>, 4194668, 0) returned error, retrying after 263.980178ms: <data/564e598c64> does not exist
Load(<data/564e598c64>, 4194668, 0) returned error, retrying after 953.286038ms: <data/564e598c64> does not exist
Load(<data/564e598c64>, 4194668, 0) returned error, retrying after 657.52647ms: <data/564e598c64> does not exist
Load(<data/564e598c64>, 4194668, 0) returned error, retrying after 2.214694858s: <data/564e598c64> does not exist
Load(<data/564e598c64>, 4194668, 0) returned error, retrying after 2.268782263s: <data/564e598c64> does not exist
Load(<data/564e598c64>, 4194668, 0) returned error, retrying after 2.996493218s: <data/564e598c64> does not exist
Load(<data/564e598c64>, 4194668, 0) returned error, retrying after 6.321174582s: <data/564e598c64> does not exist
Load(<data/564e598c64>, 4194668, 0) returned error, retrying after 6.098376127s: <data/564e598c64> does not exist
Load(<data/564e598c64>, 4194668, 0) returned error, retrying after 15.568280807s: <data/564e598c64> does not exist
Load(<data/564e598c64>, 4194668, 0) returned error, retrying after 17.428398734s: <data/564e598c64> does not exist
pack 564e598c644b4e3acf062835647ef3bf7b81502b0dc3ac29e7e74c1ad55ca999 failed to download: StreamPack: <data/564e598c64> does not exist
Load(<data/d7d3766a3c>, 4620876, 0) returned error, retrying after 718.603078ms: <data/d7d3766a3c> does not exist
Load(<data/d7d3766a3c>, 4620876, 0) returned error, retrying after 853.651411ms: <data/d7d3766a3c> does not exist
Load(<data/d7d3766a3c>, 4620876, 0) returned error, retrying after 1.477679361s: <data/d7d3766a3c> does not exist
Load(<data/d7d3766a3c>, 4620876, 0) returned error, retrying after 2.422510548s: <data/d7d3766a3c> does not exist
Load(<data/d7d3766a3c>, 4620876, 0) returned error, retrying after 1.561078081s: <data/d7d3766a3c> does not exist
Load(<data/d7d3766a3c>, 4620876, 0) returned error, retrying after 3.110752502s: <data/d7d3766a3c> does not exist
Load(<data/d7d3766a3c>, 4620876, 0) returned error, retrying after 3.663268748s: <data/d7d3766a3c> does not exist
Load(<data/d7d3766a3c>, 4620876, 0) returned error, retrying after 9.517433985s: <data/d7d3766a3c> does not exist
Load(<data/d7d3766a3c>, 4620876, 0) returned error, retrying after 14.267848553s: <data/d7d3766a3c> does not exist
Load(<data/d7d3766a3c>, 4620876, 0) returned error, retrying after 22.67087717s: <data/d7d3766a3c> does not exist
pack d7d3766a3ca5ec9db8ee5349071751be05962e1ddb19fe3f1ecdce50a25d238a failed to download: StreamPack: <data/d7d3766a3c> does not exist
rclone: 2023/09/28 17:24:22 ERROR : data/09/094b33d31d8eb9044ff51a19333c81620b65379909c98056daf6468faa9da7a4: Didn't finish writing GET request (wrote 524288/4620876 bytes): unexpected EOF
Load(<data/094b33d31d>, 4620876, 0) returned error, retrying after 590.251555ms: ReadFull: unexpected EOF
Load(<data/094b33d31d>, 4620876, 0) operation successful after 1 retries
Load(<data/fa83dafd6b>, 6569179, 0) returned error, retrying after 506.227459ms: <data/fa83dafd6b> does not exist
Load(<data/fa83dafd6b>, 6569179, 0) returned error, retrying after 697.960784ms: <data/fa83dafd6b> does not exist
Load(<data/fa83dafd6b>, 6569179, 0) returned error, retrying after 901.334837ms: <data/fa83dafd6b> does not exist
Load(<data/fa83dafd6b>, 6569179, 0) returned error, retrying after 2.134415943s: <data/fa83dafd6b> does not exist
Load(<data/fa83dafd6b>, 6569179, 0) returned error, retrying after 3.207297455s: <data/fa83dafd6b> does not exist
Load(<data/fa83dafd6b>, 6569179, 0) returned error, retrying after 4.319438752s: <data/fa83dafd6b> does not exist
Load(<data/fa83dafd6b>, 6569179, 0) returned error, retrying after 7.976548371s: <data/fa83dafd6b> does not exist
Load(<data/fa83dafd6b>, 6569179, 0) returned error, retrying after 4.678708479s: <data/fa83dafd6b> does not exist
Load(<data/fa83dafd6b>, 6569179, 0) returned error, retrying after 12.349210342s: <data/fa83dafd6b> does not exist
Load(<data/fa83dafd6b>, 6569179, 0) returned error, retrying after 9.742702561s: <data/fa83dafd6b> does not exist
pack fa83dafd6b49dd762415dc619404ac3296224f5750264d482b8d580116f3f4b2 failed to download: StreamPack: <data/fa83dafd6b> does not exist
Load(<data/2dea030b90>, 4843409, 0) returned error, retrying after 354.502188ms: <data/2dea030b90> does not exist
Load(<data/2dea030b90>, 4843409, 0) returned error, retrying after 438.853812ms: <data/2dea030b90> does not exist
Load(<data/2dea030b90>, 4843409, 0) returned error, retrying after 1.251106582s: <data/2dea030b90> does not exist
Load(<data/2dea030b90>, 4843409, 0) returned error, retrying after 2.450905776s: <data/2dea030b90> does not exist
Load(<data/2dea030b90>, 4843409, 0) returned error, retrying after 2.988084451s: <data/2dea030b90> does not exist
Load(<data/2dea030b90>, 4843409, 0) returned error, retrying after 3.471035099s: <data/2dea030b90> does not exist
Load(<data/2dea030b90>, 4843409, 0) returned error, retrying after 8.251946255s: <data/2dea030b90> does not exist
Load(<data/2dea030b90>, 4843409, 0) returned error, retrying after 8.963717301s: <data/2dea030b90> does not exist
Load(<data/2dea030b90>, 4843409, 0) returned error, retrying after 6.803697865s: <data/2dea030b90> does not exist
Load(<data/2dea030b90>, 4843409, 0) returned error, retrying after 28.708992951s: <data/2dea030b90> does not exist
pack 2dea030b90d720c6d926368d74e6e0051bc4bdb9110d6104a8c61fe21cd648c2 failed to download: StreamPack: <data/2dea030b90> does not exist
Load(<data/fe2987fb6a>, 4577707, 0) returned error, retrying after 515.818437ms: <data/fe2987fb6a> does not exist
Load(<data/fe2987fb6a>, 4577707, 0) returned error, retrying after 1.059693256s: <data/fe2987fb6a> does not exist
Load(<data/fe2987fb6a>, 4577707, 0) returned error, retrying after 674.175215ms: <data/fe2987fb6a> does not exist
Load(<data/fe2987fb6a>, 4577707, 0) returned error, retrying after 2.20055658s: <data/fe2987fb6a> does not exist
Load(<data/fe2987fb6a>, 4577707, 0) returned error, retrying after 2.83880298s: <data/fe2987fb6a> does not exist
Load(<data/fe2987fb6a>, 4577707, 0) returned error, retrying after 3.874898701s: <data/fe2987fb6a> does not exist
Load(<data/fe2987fb6a>, 4577707, 0) returned error, retrying after 6.451817977s: <data/fe2987fb6a> does not exist
Load(<data/fe2987fb6a>, 4577707, 0) returned error, retrying after 4.92616409s: <data/fe2987fb6a> does not exist
Load(<data/fe2987fb6a>, 4577707, 0) returned error, retrying after 9.029063442s: <data/fe2987fb6a> does not exist
Load(<data/fe2987fb6a>, 4577707, 0) returned error, retrying after 16.148926215s: <data/fe2987fb6a> does not exist
pack fe2987fb6a283589175c6dec1ea8d53e0fd4a0390b79ad5ab60279d8cdd07a86 failed to download: StreamPack: <data/fe2987fb6a> does not exist
Load(<data/2fd6c10873>, 4661811, 0) returned error, retrying after 607.301238ms: <data/2fd6c10873> does not exist
Load(<data/2fd6c10873>, 4661811, 0) returned error, retrying after 377.914165ms: <data/2fd6c10873> does not exist
Load(<data/2fd6c10873>, 4661811, 0) returned error, retrying after 1.454486481s: <data/2fd6c10873> does not exist
Load(<data/2fd6c10873>, 4661811, 0) returned error, retrying after 958.744263ms: <data/2fd6c10873> does not exist
Load(<data/2fd6c10873>, 4661811, 0) returned error, retrying after 2.873316872s: <data/2fd6c10873> does not exist
Load(<data/2fd6c10873>, 4661811, 0) returned error, retrying after 4.556138278s: <data/2fd6c10873> does not exist
Load(<data/2fd6c10873>, 4661811, 0) returned error, retrying after 3.010097381s: <data/2fd6c10873> does not exist
Load(<data/2fd6c10873>, 4661811, 0) returned error, retrying after 4.357059873s: <data/2fd6c10873> does not exist
Load(<data/2fd6c10873>, 4661811, 0) returned error, retrying after 18.686588598s: <data/2fd6c10873> does not exist
Load(<data/2fd6c10873>, 4661811, 0) returned error, retrying after 13.887651534s: <data/2fd6c10873> does not exist
pack 2fd6c1087330362eb429d7b4b167e5081d034bc3dfa34aaa6f149317642b803d failed to download: StreamPack: <data/2fd6c10873> does not exist
[35:41:00] 100.00%  167414 / 167414 packs

Fatal: repository contains errors

Thanks a lot for your support!!!

MichaelEischer · September 30, 2023, 11:39am

As check all packs does not report an error, this means that all necessary files show up when listing the stored files. However, later attempts to download them sometimes fail with pack [...] failed to download. Restic made ten attempts to download the file, however, pCloud repeatedly failed to return the file. If you can manually download these files, then your repository is probably intact, however, you might run into problems when trying to restore data.

I don’t think there’s much we can do in restic to work around this problem. (except maybe allowing a few additional retries). The only proper way to fix this is on pCloud’s side. (If I read the above logs correctly, the failing files change between different check runs?)

T-6 · September 30, 2023, 2:07pm

I have logs of 6 runs of restic check --read-data. Not all of them are from completed runs. Some packs show up several times (up to 4 times), others only once.

What about the sha256 sum of these files which were not the same as the filename.
If I download the files and check the sum, and the sum does not match the filename, would this indicate that the files are corrupted (for whatever reason)?

T-6 · September 30, 2023, 2:37pm

Just tried to download the 11 files with issues. 8 of them could actually not be downloaded via the browser. The other three downloaded without issue and the hash matches the filename.

None of these three files caused issues during the last run of restic check.

I contacted the pCloud support to find out why the other 8 files could not be downloaded.

T-6 · October 3, 2023, 12:45pm

Reply from pCloud support:

The file is temporary unavailable due to issues with the storage servers. You should simply wait so this hardware issue can be fixed and the files will be available at some point.

Not really satisfactory.

MichaelEischer · October 3, 2023, 3:15pm

Thanks for reporting back, these hardware issues really take a long time to get fixed.

One maybe interesting fact to add here is that one has to distinguish between Durability (data is stored correctly) and Availability (data can be retrieved right now). Durability is usually a 99.99999999% (or similar) number, whereas availability is 99.9% at best. See also Cloud Storage Durability vs. Availability: What Are the Differences? for a more detailed explanation.

T-6 · December 3, 2023, 10:04am

Finally some progress. The packs, which have been reported missing by check are available again. pCloud fixed the problem after almost 2 month. Will now run a complete check again to see if all packs are available in good.

rawtaz · December 3, 2023, 1:33pm

Sounds like a good example of why one shouldn’t use cheap storage services

kapitainsky · December 3, 2023, 2:46pm

pCloud is actually not cheap:) $5+ per TB per month.

rawtaz · December 3, 2023, 3:05pm

You don’t call that cheap? Oh my… I think that’s very low cost storage, and that you get what you pay for. I pay €53.62/month/TB (SSD block storage on Ceph).

T-6 · December 5, 2023, 5:45pm

Check finished without issues. So I have confidence in Durability. Most important to me as this is only the offsite backup of the 3-2-1 backup strategy.