Repairing repository with several corrupted packs

Getting Help
1

Hello! I’ve been using restic for about a year doing manual backups every month or so. Yesterday I wanted to delete older snapshots with forget but got

collecting packs for deletion and repacking
pack 59cdbff3: calculated size 17865628 does not match real size 17832860
Run ‘restic repair index’.
[0:39] 34.67% 22288 / 64290 packs processed
Fatal: pack size does not match calculated size from index

And so I ran restic repair index and got the following:

check all packs
pack e4c758ad77726b80c390fc06870733605553cfc3ed94d90519ce23919883bccc: unexpected file size: got 17220228, expected 17252996
pack e6849cee43d281bb854f144695ae8ec5168cf76750afc11d14da184fee4bce19: unexpected file size: got 17875658, expected 17908426
pack def3e94368415031654248885f15316f4c448e46b00a9a30a4da6090bb029591: unexpected file size: got 17226825, expected 17259593
pack 59cdbff3f00b0c00e716a77115944f785f6d9f7203a0e4cde42bef5d95465fd8: unexpected file size: got 17832860, expected 17865628
check snapshots, trees and blobs
[0:07] 100.00% 4 / 4 snapshots
Fatal: repository contains errors

I checked what files are affected with find --pack and I still have originals of these files ready to be backed up but I don’t understand how can I remove these packs for now.

Looking through troubleshooting docs I thought that repair snapshots --forget should do the trick and create new snapshot without corrupted packs/files but in my case it did nothing.

[0:00] 100.00% 17 / 17 index files loaded
snapshot 691d05c3 of [/data] at 2024-01-09 05:12:37.484650267 +0000 UTC)
snapshot c2d0714b of [/mnt/volumes] at 2024-03-11 09:13:17.501441537 +0700 +07)
snapshot 978e3d81 of [/mnt/volumes] at 2024-03-10 23:19:46.214827111 +0700 +07)
snapshot 8f05e76a of [/mnt/volumes] at 2024-03-09 16:10:46.798878091 +0700 +07)
no snapshots were modified

What can I do now to remove corrupted packs and backup affected files again. In worst case I could just create new repository from scratch but I’d like to avoid that since backing up to my remote could take several days.

2

restic version 0.16.4:

  1. restic repair packs packIDs.... Restic will salvage bad packs.
  2. Don’t run repair snapshots --forget at this time. Instead, run restic backup. Restic will find that some packs are missing and re-store them. (Provided that you still have originals of these files)
3

Seems like I’m having more problems…

Summary

tumist@Tumist-MacBook ~ % restic -r rclone:yandexdisk-tumist76:/Backups/restic repair packs 59cdbff3f00b0c00e716a77115944f785f6d9f7203a0e4cde42bef5d95465fd8 def3e94368415031654248885f15316f4c448e46b00a9a30a4da6090bb029591 e4c758ad77726b80c390fc06870733605553cfc3ed94d90519ce23919883bccc e6849cee43d281bb854f144695ae8ec5168cf76750afc11d14da184fee4bce19

enter password for repository:
repository 927f4fa8 opened (version 2, compression level auto)
found 1 old cache directories in /Users/tumist/Library/Caches/restic, run restic cache --cleanup to remove them
[0:00] 100.00% 17 / 17 index files loaded
saving backup copies of pack files in current folder
Fatal: open pack-59cdbff3f00b0c00e716a77115944f785f6d9f7203a0e4cde42bef5d95465fd8: file exists
tumist@Tumist-MacBook ~ % restic -r rclone:yandexdisk-tumist76:/Backups/restic repair packs 59cdbff3f00b0c00e716a77115944f785f6d9f7203a0e4cde42bef5d95465fd8 def3e94368415031654248885f15316f4c448e46b00a9a30a4da6090bb029591 e4c758ad77726b80c390fc06870733605553cfc3ed94d90519ce23919883bccc e6849cee43d281bb854f144695ae8ec5168cf76750afc11d14da184fee4bce19
enter password for repository:
repository 927f4fa8 opened (version 2, compression level auto)
found 1 old cache directories in /Users/tumist/Library/Caches/restic, run restic cache --cleanup to remove them
[0:00] 100.00% 17 / 17 index files loaded
saving backup copies of pack files in current folder
salvaging intact data from specified pack files
failed to load blob 2b985b732a11a67812aabd576d2660652e33fd091812e06f861b246aed312e9a: decrypting blob 2b985b732a11a67812aabd576d2660652e33fd091812e06f861b246aed312e9a failed: ciphertext verification failed
failed to load blob cd1b6c909b83a393a6a62445f6191019bc413221cc0f91dbaf363ea35909b37d: decrypting blob cd1b6c909b83a393a6a62445f6191019bc413221cc0f91dbaf363ea35909b37d failed: ciphertext verification failed
failed to load blob a2fca095de30c91daa704da987a04ed463bca013e64be50559a34a1bbf80050f: decrypting blob a2fca095de30c91daa704da987a04ed463bca013e64be50559a34a1bbf80050f failed: ciphertext verification failed
failed to load blob d2f37b8d1916b2cf1a13e5c1649fb868eb112179bae4361471d19a68fb581d37: decrypting blob d2f37b8d1916b2cf1a13e5c1649fb868eb112179bae4361471d19a68fb581d37 failed: ciphertext verification failed
failed to load blob afcda6877b159ff1da22c0c04e3b3185a3a4fa80ffd42357d51c332800a92f64: decrypting blob afcda6877b159ff1da22c0c04e3b3185a3a4fa80ffd42357d51c332800a92f64 failed: ciphertext verification failed

failed to load blob 4d3c83f1d338dadf6ac0008c8c995f8507289a770f6a26e96bfc163ed924e369: decrypting blob 4d3c83f1d338dadf6ac0008c8c995f8507289a770f6a26e96bfc163ed924e369 failed: ciphertext verification failed
failed to load blob d931637785118915080edddefcbd7d48f3b06126b484c3783495fb2719c32e1b: decrypting blob d931637785118915080edddefcbd7d48f3b06126b484c3783495fb2719c32e1b failed: ciphertext verification failed
failed to load blob ad0feafe6c415d37d6ed8187068c6440566069c114f8508f1a1f1f7f0fa3bd59: decrypting blob ad0feafe6c415d37d6ed8187068c6440566069c114f8508f1a1f1f7f0fa3bd59 failed: ciphertext verification failed
failed to load blob c22ae8a5b59ab405a480016b7931e962de6e6c6b476bfda84467fb2c016b6edc: decrypting blob c22ae8a5b59ab405a480016b7931e962de6e6c6b476bfda84467fb2c016b6edc failed: ciphertext verification failed
failed to load blob 32cc6fec880aeea3c2b0ec9df6dfe994533f18ee8319d5a363935799c8f04234: decrypting blob 32cc6fec880aeea3c2b0ec9df6dfe994533f18ee8319d5a363935799c8f04234 failed: ciphertext verification failed
failed to load blob befdbdbe1e42e7b98ee764ef41acdf876ddede09d2e7d9136fdbb97c55cdbeb4: decrypting blob befdbdbe1e42e7b98ee764ef41acdf876ddede09d2e7d9136fdbb97c55cdbeb4 failed: ciphertext verification failed
failed to load blob 5bc6ed56a17906180d82a218b287870c7eb220e03b8ba46ad2d8f38c7cc17367: decrypting blob 5bc6ed56a17906180d82a218b287870c7eb220e03b8ba46ad2d8f38c7cc17367 failed: ciphertext verification failed
failed to load blob 033498be4d0a28e8d927c12e6c1a4a6a7ce0b5dfcfe71268e1ff3ea34acc8fd9: decrypting blob 033498be4d0a28e8d927c12e6c1a4a6a7ce0b5dfcfe71268e1ff3ea34acc8fd9 failed: ciphertext verification failed
Load(<data/e4c758ad77>, 17252386, 0) returned error, retrying after 662.004158ms: ReadFull: unexpected EOF
Load(<data/e4c758ad77>, 17252386, 0) returned error, retrying after 583.220069ms: ReadFull: unexpected EOF
Load(<data/e4c758ad77>, 17252386, 0) returned error, retrying after 905.714822ms: ReadFull: unexpected EOF
Load(<data/e4c758ad77>, 17252386, 0) returned error, retrying after 1.917380181s: ReadFull: unexpected EOF
Load(<data/e4c758ad77>, 17252386, 0) returned error, retrying after 1.310928903s: ReadFull: unexpected EOF
Load(<data/e4c758ad77>, 17252386, 0) returned error, retrying after 5.659748334s: ReadFull: unexpected EOF
Load(<data/e4c758ad77>, 17252386, 0) returned error, retrying after 5.313781898s: ReadFull: unexpected EOF
Load(<data/e4c758ad77>, 17252386, 0) returned error, retrying after 8.10452268s: ReadFull: unexpected EOF
Load(<data/e4c758ad77>, 17252386, 0) returned error, retrying after 18.141893025s: ReadFull: unexpected EOF
signal interrupt received, cleaning up
[1:11] 0.00% 0 / 4 pack files
Fatal: StreamPack: context canceled
rclone: 2024/03/11 11:37:24 ERROR : data/e4/e4c758ad77726b80c390fc06870733605553cfc3ed94d90519ce23919883bccc: Didn’t finish writing GET request (wrote 6593024/17220228 bytes): context canceled

In the end I’ve just cancelled the process with ctrl+c

4

I’m not sure, but it seems that repair packs is doing its work…

5

Dunno, the troubleshooting page says

If the output contains warnings that the ciphertext verification failed for some blobs in the repository, then please ask for help in the forum or our IRC channel. These errors are often caused by hardware problems which must be investigated and fixed. Otherwise, the backup will be damaged again and again.

I’ll try to run the command again and wait for an hour to see if this changes anything.

6

Well, could have waited a little longer for a fatal error :slight_smile:

Summary

tumist@Tumist-MacBook ~ % restic -r rclone:yandexdisk-tumist76:/Backups/restic repair packs 59cdbff3f00b0c00e716a77115944f785f6d9f7203a0e4cde42bef5d95465fd8 def3e94368415031654248885f15316f4c448e46b00a9a30a4da6090bb029591 e4c758ad77726b80c390fc06870733605553cfc3ed94d90519ce23919883bccc e6849cee43d281bb854f144695ae8ec5168cf76750afc11d14da184fee4bce19

enter password for repository:

repository 927f4fa8 opened (version 2, compression level auto)

found 1 old cache directories in /Users/tumist/Library/Caches/restic, run restic cache --cleanup to remove them

[0:00] 100.00% 17 / 17 index files loaded

saving backup copies of pack files in current folder

salvaging intact data from specified pack files

failed to load blob 2b985b732a11a67812aabd576d2660652e33fd091812e06f861b246aed312e9a: decrypting blob 2b985b732a11a67812aabd576d2660652e33fd091812e06f861b246aed312e9a failed: ciphertext verification failed

failed to load blob cd1b6c909b83a393a6a62445f6191019bc413221cc0f91dbaf363ea35909b37d: decrypting blob cd1b6c909b83a393a6a62445f6191019bc413221cc0f91dbaf363ea35909b37d failed: ciphertext verification failed

failed to load blob a2fca095de30c91daa704da987a04ed463bca013e64be50559a34a1bbf80050f: decrypting blob a2fca095de30c91daa704da987a04ed463bca013e64be50559a34a1bbf80050f failed: ciphertext verification failed

failed to load blob d2f37b8d1916b2cf1a13e5c1649fb868eb112179bae4361471d19a68fb581d37: decrypting blob d2f37b8d1916b2cf1a13e5c1649fb868eb112179bae4361471d19a68fb581d37 failed: ciphertext verification failed

failed to load blob afcda6877b159ff1da22c0c04e3b3185a3a4fa80ffd42357d51c332800a92f64: decrypting blob afcda6877b159ff1da22c0c04e3b3185a3a4fa80ffd42357d51c332800a92f64 failed: ciphertext verification failed

failed to load blob 4d3c83f1d338dadf6ac0008c8c995f8507289a770f6a26e96bfc163ed924e369: decrypting blob 4d3c83f1d338dadf6ac0008c8c995f8507289a770f6a26e96bfc163ed924e369 failed: ciphertext verification failed

failed to load blob d931637785118915080edddefcbd7d48f3b06126b484c3783495fb2719c32e1b: decrypting blob d931637785118915080edddefcbd7d48f3b06126b484c3783495fb2719c32e1b failed: ciphertext verification failed

failed to load blob ad0feafe6c415d37d6ed8187068c6440566069c114f8508f1a1f1f7f0fa3bd59: decrypting blob ad0feafe6c415d37d6ed8187068c6440566069c114f8508f1a1f1f7f0fa3bd59 failed: ciphertext verification failed

failed to load blob c22ae8a5b59ab405a480016b7931e962de6e6c6b476bfda84467fb2c016b6edc: decrypting blob c22ae8a5b59ab405a480016b7931e962de6e6c6b476bfda84467fb2c016b6edc failed: ciphertext verification failed

failed to load blob 32cc6fec880aeea3c2b0ec9df6dfe994533f18ee8319d5a363935799c8f04234: decrypting blob 32cc6fec880aeea3c2b0ec9df6dfe994533f18ee8319d5a363935799c8f04234 failed: ciphertext verification failed

failed to load blob befdbdbe1e42e7b98ee764ef41acdf876ddede09d2e7d9136fdbb97c55cdbeb4: decrypting blob befdbdbe1e42e7b98ee764ef41acdf876ddede09d2e7d9136fdbb97c55cdbeb4 failed: ciphertext verification failed

failed to load blob 5bc6ed56a17906180d82a218b287870c7eb220e03b8ba46ad2d8f38c7cc17367: decrypting blob 5bc6ed56a17906180d82a218b287870c7eb220e03b8ba46ad2d8f38c7cc17367 failed: ciphertext verification failed

failed to load blob 033498be4d0a28e8d927c12e6c1a4a6a7ce0b5dfcfe71268e1ff3ea34acc8fd9: decrypting blob 033498be4d0a28e8d927c12e6c1a4a6a7ce0b5dfcfe71268e1ff3ea34acc8fd9 failed: ciphertext verification failed

Load(<data/e4c758ad77>, 17252386, 0) returned error, retrying after 594.327661ms: ReadFull: unexpected EOF

Load(<data/e4c758ad77>, 17252386, 0) returned error, retrying after 858.70775ms: ReadFull: unexpected EOF

Load(<data/e4c758ad77>, 17252386, 0) returned error, retrying after 861.345505ms: ReadFull: unexpected EOF

Load(<data/e4c758ad77>, 17252386, 0) returned error, retrying after 900.063994ms: ReadFull: unexpected EOF

Load(<data/e4c758ad77>, 17252386, 0) returned error, retrying after 1.67004029s: ReadFull: unexpected EOF

Load(<data/e4c758ad77>, 17252386, 0) returned error, retrying after 4.93014111s: ReadFull: unexpected EOF

Load(<data/e4c758ad77>, 17252386, 0) returned error, retrying after 5.569220216s: ReadFull: unexpected EOF

Load(<data/e4c758ad77>, 17252386, 0) returned error, retrying after 8.806305984s: ReadFull: unexpected EOF

Load(<data/e4c758ad77>, 17252386, 0) returned error, retrying after 16.31386574s: ReadFull: unexpected EOF

Load(<data/e4c758ad77>, 17252386, 0) returned error, retrying after 17.169995271s: ReadFull: unexpected EOF

[1:34] 0.00% 0 / 4 pack files

Fatal: StreamPack: ReadFull: unexpected EOF

7

Looks like you have problems with network connectivity. Cloud provider throttling your operations? What is your remote?

8

Yeah, It could be throttled for sure. It’s Yandex Disk using rclone.

9

add --tpslimit 10 --tpslimit-burst 0 to your rclone command. If still see errors decrease tpslimit

10

Maybe I’m wrong but looking at rclone output it seems to me like data has been successfully downloaded from remote but the error is still the same:

Output

tumist@Tumist-MacBook ~ % restic -r rclone:yandexdisk-tumist76:/Backups/restic -o rclone.args=“serve restic --stdio --tpslimit 10 --tpslimit-burst 0 --verbose” repair packs 59cdbff3f00b0c00e716a77115944f785f6d9f7203a0e4cde42bef5d95465fd8 def3e94368415031654248885f15316f4c448e46b00a9a30a4da6090bb029591 e4c758ad77726b80c390fc06870733605553cfc3ed94d90519ce23919883bccc e6849cee43d281bb854f144695ae8ec5168cf76750afc11d14da184fee4bce19

rclone: 2024/03/11 15:27:13 INFO : Starting transaction limiter: max 10 transactions/s with burst 1

enter password for repository:

repository 927f4fa8 opened (version 2, compression level auto)

found 1 old cache directories in /Users/tumist/Library/Caches/restic, run restic cache --cleanup to remove them

[0:01] 100.00% 17 / 17 index files loaded

saving backup copies of pack files in current folder

salvaging intact data from specified pack files

failed to load blob 2b985b732a11a67812aabd576d2660652e33fd091812e06f861b246aed312e9a: decrypting blob 2b985b732a11a67812aabd576d2660652e33fd091812e06f861b246aed312e9a failed: ciphertext verification failed

failed to load blob cd1b6c909b83a393a6a62445f6191019bc413221cc0f91dbaf363ea35909b37d: decrypting blob cd1b6c909b83a393a6a62445f6191019bc413221cc0f91dbaf363ea35909b37d failed: ciphertext verification failed

failed to load blob a2fca095de30c91daa704da987a04ed463bca013e64be50559a34a1bbf80050f: decrypting blob a2fca095de30c91daa704da987a04ed463bca013e64be50559a34a1bbf80050f failed: ciphertext verification failed

failed to load blob d2f37b8d1916b2cf1a13e5c1649fb868eb112179bae4361471d19a68fb581d37: decrypting blob d2f37b8d1916b2cf1a13e5c1649fb868eb112179bae4361471d19a68fb581d37 failed: ciphertext verification failed

failed to load blob afcda6877b159ff1da22c0c04e3b3185a3a4fa80ffd42357d51c332800a92f64: decrypting blob afcda6877b159ff1da22c0c04e3b3185a3a4fa80ffd42357d51c332800a92f64 failed: ciphertext verification failed

failed to load blob 4d3c83f1d338dadf6ac0008c8c995f8507289a770f6a26e96bfc163ed924e369: decrypting blob 4d3c83f1d338dadf6ac0008c8c995f8507289a770f6a26e96bfc163ed924e369 failed: ciphertext verification failed

failed to load blob d931637785118915080edddefcbd7d48f3b06126b484c3783495fb2719c32e1b: decrypting blob d931637785118915080edddefcbd7d48f3b06126b484c3783495fb2719c32e1b failed: ciphertext verification failed

failed to load blob ad0feafe6c415d37d6ed8187068c6440566069c114f8508f1a1f1f7f0fa3bd59: decrypting blob ad0feafe6c415d37d6ed8187068c6440566069c114f8508f1a1f1f7f0fa3bd59 failed: ciphertext verification failed

failed to load blob c22ae8a5b59ab405a480016b7931e962de6e6c6b476bfda84467fb2c016b6edc: decrypting blob c22ae8a5b59ab405a480016b7931e962de6e6c6b476bfda84467fb2c016b6edc failed: ciphertext verification failed

failed to load blob 32cc6fec880aeea3c2b0ec9df6dfe994533f18ee8319d5a363935799c8f04234: decrypting blob 32cc6fec880aeea3c2b0ec9df6dfe994533f18ee8319d5a363935799c8f04234 failed: ciphertext verification failed

failed to load blob befdbdbe1e42e7b98ee764ef41acdf876ddede09d2e7d9136fdbb97c55cdbeb4: decrypting blob befdbdbe1e42e7b98ee764ef41acdf876ddede09d2e7d9136fdbb97c55cdbeb4 failed: ciphertext verification failed

failed to load blob 5bc6ed56a17906180d82a218b287870c7eb220e03b8ba46ad2d8f38c7cc17367: decrypting blob 5bc6ed56a17906180d82a218b287870c7eb220e03b8ba46ad2d8f38c7cc17367 failed: ciphertext verification failed

failed to load blob 033498be4d0a28e8d927c12e6c1a4a6a7ce0b5dfcfe71268e1ff3ea34acc8fd9: decrypting blob 033498be4d0a28e8d927c12e6c1a4a6a7ce0b5dfcfe71268e1ff3ea34acc8fd9 failed: ciphertext verification failed

Load(<data/e4c758ad77>, 17252386, 0) returned error, retrying after 308.763051ms: ReadFull: unexpected EOF

Load(<data/e4c758ad77>, 17252386, 0) returned error, retrying after 915.209173ms: ReadFull: unexpected EOF

Load(<data/e4c758ad77>, 17252386, 0) returned error, retrying after 690.07031ms: ReadFull: unexpected EOF

Load(<data/e4c758ad77>, 17252386, 0) returned error, retrying after 1.773755676s: ReadFull: unexpected EOF

Load(<data/e4c758ad77>, 17252386, 0) returned error, retrying after 2.639830604s: ReadFull: unexpected EOF

rclone: 2024/03/11 15:28:14 INFO :

rclone: Transferred: 164.727 MiB / 164.727 MiB, 100%, 3.389 MiB/s, ETA 0s

rclone: Transferred: 28 / 28, 100%

rclone: Elapsed time: 1m0.6s

rclone:

Load(<data/e4c758ad77>, 17252386, 0) returned error, retrying after 2.361044299s: ReadFull: unexpected EOF

Load(<data/e4c758ad77>, 17252386, 0) returned error, retrying after 2.985539035s: ReadFull: unexpected EOF

Load(<data/e4c758ad77>, 17252386, 0) returned error, retrying after 10.683858923s: ReadFull: unexpected EOF

Load(<data/e4c758ad77>, 17252386, 0) returned error, retrying after 10.521005477s: ReadFull: unexpected EOF

Load(<data/e4c758ad77>, 17252386, 0) returned error, retrying after 10.156393595s: ReadFull: unexpected EOF

[1:22] 0.00% 0 / 4 pack files

Fatal: StreamPack: ReadFull: unexpected EOF

11

Unfortunately Yandex is not very reliable - rclone forum is full of problem stories, e.g.:

1 Like
12

Fair enough. Maybe I’ll move somewhere else after my subscription ends. Anyway, is there a way to simply remove these corrupted packs from the snapshots other than restic repair packs?

13

I don’t know if this is an appropriate way, but I just found it works like repair packs. Just for reference.

  1. Delete those corrupted pack files manually
  2. restic repair index. Restic will find that some packs are missing and remove them from index.
  3. restic backup. Restic will find that some files are missing thanks to the index updated in step 2 and re-store them to the repository. Provided that you still have all original files that are affected by the corrupted packs, after this step, all your snapshots will become healthy again.
14

By the way, how do you give your terminal output in the post a title like ‘Output’ and ‘Summary’ and make it a dropdown?

15
  1. Select text what you want to hide.
  2. Press “gear” icon in the editing toolbar
  3. Select “Hide details”.

Selected text will be wrapped in details tag and you’ll be able to change the title.

2 Likes
16

After several hours of pruning and checking, it finally completed and everything’s fine now, thanks! Maybe it’s not a proper way to repair repo but if it works, it works :slight_smile:

2 Likes
17

repair packs will be able to handle truncated pack files once restic 0.17.0 is released.

That error is expected with restic 0.16.4 and truncated pack files.

Those steps are ok (for now). Although, once repair packs can handle truncated files, please use that command.