Today I do a daily backup on my laptop and after a prune operation I see this error:
root@internal:~# restic -r s3:<my-bucket> prune --password-file "/my-secret.txt"
repository 02c1c5d6 opened (version 2, compression level auto)
loading indexes...
[0:00] 100.00% 2 / 2 index files loaded
loading all snapshots...
finding data that is still in use for 10 snapshots
[0:01] 100.00% 10 / 10 snapshots
searching used packs...
collecting packs for deletion and repacking
[0:02] 100.00% 2998 / 2998 packs processed
to repack: 20155 blobs / 41.386 MiB
this removes: 280 blobs / 30.108 MiB
to delete: 179 blobs / 430.218 MiB
total prune: 459 blobs / 460.325 MiB
not yet compressed: 15.412 GiB
remaining: 97254 blobs / 15.413 GiB
unused size after prune: 788.247 MiB (4.99% of remaining size)
repacking packs
[0:02] 50.00% 3 / 6 packs repacked
Fatal: StreamPack: ciphertext verification failed
root@internal:~#
apparently with no error in the forget or backup command.
after this error, I run immediately the check of the data and this is the result:
root@internal:~# restic -r s3:<my-bucket> check --read-data --password-file "/my-secret.txt"
using temporary cache in /tmp/restic-check-cache-359632852
repository 02c1c5d6 opened (version 2, compression level auto)
created new cache in /tmp/restic-check-cache-359632852
create exclusive lock for repository
load indexes
[0:06] 100.00% 2 / 2 index files loaded
check all packs
check snapshots, trees and blobs
[0:18] 100.00% 10 / 10 snapshots
read all data
[40:51] 100.00% 2998 / 2998 packs
no errors were found
root@internal:~#
My restic version is (installed manually from github releases):
root@internal:~# restic version
restic 0.16.4 compiled with go1.21.6 on linux/amd64
As check --read-data states that your repository is intact, that means that the prune error was likely a temporary issue (likely a bitflip in hardware). I assume that if you run prune again, that the error does not show up again?
You could run a CPU / memory stress test on your laptop to make sure that the hardware is working reliably.
Hmm, that error could originate from the same pack file as before. Unfortunately the error message does not include the affected pack file (will be fixed by https://github.com/restic/restic/pull/4701). Please run restic with the environment variable DEBUG_LOG=restic.log. The relevant part of the log starts with the first line containing repository.streamPackPart.
What still doesn’t make sense for me is that check --read-data did not detect any data corruption. So let’s try the following: please look up the location of the cache folder used by restic (see restic cache) and rename that folder. Does prune work afterwards (it should still yield the same error as before, but let’s give it a try).
I usually use prime95 which has a stress test mode.
I have also removed the cache folder and redo a check, but after a new backup, forget and prune (with the same error) no errors was found:
root@internal:~# restic -r s3:<host> check --read-data --password-file "/my-secret.txt"
using temporary cache in /tmp/restic-check-cache-3203237224
repository 02c1c5d6 opened (version 2, compression level auto)
created new cache in /tmp/restic-check-cache-3203237224
create exclusive lock for repository
load indexes
[0:01] 100.00% 5 / 5 index files loaded
check all packs
check snapshots, trees and blobs
[0:03] 100.00% 10 / 10 snapshots
read all data
[16:16] 100.00% 3013 / 3013 packs
no errors were found
As check --read-data does not report errors but prune does, let’s dig into the main difference here: check downloads the full file, whereas prune only downloads the relevant section (see the Range header). Apparently that S3 storage is unable to correctly handle the Range header.
To see whether that suspicion is warranted, please download the whole repository to a local folder and then run prune on that copy. If that works, then this is clearly a problem of the S3 storage.
If that S3 storage is indeed unable to correctly handle the Range header, that would also perfectly explain the errors from the Github issue. The pack header is the last part of a file, so restic only retrieves that part. Apparently, the returned data is either the wrong part of the file or just garbage and hence triggers the incomplete pack file in older restic versions.
Based on the other errors in the Github issue, I’d start worrying whether you’ll actually be able to restore your data if you need to.
I think you should open ticket with your provider. You are paying for S3 compatible product - and it seems it is not. Maybe some misconfiguration on their end or load/performance issues.
Quick google tells me that it is better to stay away from them, e.g.:
prune already contains several fallback mechanisms to try to handle limited corrupted data. But if the storage backend returns garbage, then that is a bug of the storage provider, not of restic. Thus, this only can be fixed by the storage provider. We’re not going to introduce workarounds for weirdly broken storage providers.
I agree with you! My intention wasn’t to suggest introducing a workaround.
I am sorry, I mean if the prune command already has some parameter to avoid using this header.
Anyway, I have contacted the provider and it guarantees me that the range header is supported
I will try to go deeper in the argument.
Well, problem is not that range header is not supported:) It happily accepts request - but returns wrong data.
I use restic myself with couple of S3 providers - no issues at all.
Now as I understand from your post it used to work in the past - so something changed. If not restic version than something on your provider site. If you run daily backups you know when it stopped working - maybe it will give them some clue as it could be that around that date they updated some software - and it has a bug.
In my last post it worked because after contacting Cubbit support via email they immediately released a fix for the bug described in this post (I also sent them the link to this thread).
Only yesterday they sent me an email to confirm that the bug has been permanently fixed.
I thank everyone for their willingness to support me <3
