Decrypting blob failed + Unable to load tree

Getting Help
1

I need some help understanding what’s going on here and how to resolve it. Getting the errors from the first command, I started a search for the first blob so I could remove it, but the search just gives errors.

I’m using restic 16.0 on an Raspberry Pi 3 with the repository on a local USB spinning disk.

jarle@pi01:~/restic $ ./restic check 
using temporary cache in /tmp/restic-check-cache-436818787                                                                                                           repository bf16cbdf opened (version 2, compression level auto)                                                                                                       created new cache in /tmp/restic-check-cache-436818787                                                                                                               create exclusive lock for repository                                                                                                                                 load indexes                                                                                                                                                         check all packs                                                                                                                                                      check snapshots, trees and blobs                                                                                                                                     error for tree 39d4bd7d:                                                                                                                                               decrypting blob 39d4bd7d4708b99c3708d617064266932d1a3d139a252ea22f45098fee144c98 failed: nonce is invalid
error for tree 39d4bd7d:                                                                                                                                     [0/1290]  decrypting blob 39d4bd7d4708b99c3708d617064266932d1a3d139a252ea22f45098fee144c98 failed: nonce is invalid                                                          error for tree 8f1340e0:
  decrypting blob 8f1340e0ad6515f4f585178e2a45d65fdd80b37e727b622467298c3cd23051ab failed: nonce is invalid                                                          error for tree 00e2d7a1:
  decrypting blob 00e2d7a10d5e32630930702a7da3c6e3f2bb773569f520d179cbcd082d17bd7e failed: nonce is invalid                                                          error for tree e551388d:
  decrypting blob e551388d349543156bc83284791650c041f1e2fcc6ccc064418c88990a9854ab failed: nonce is invalid                                                          error for tree 552164d1:
  decrypting blob 552164d11c6c331d569e9560ab8fdeb51ea80f015c586b2080d357ac546ca4d1 failed: nonce is invalid                                                          [7:20] 100.00%  709 / 709 snapshots
Fatal: repository contains errors

jarle@pi01:~/restic $ ./restic find --blob 39d4bd7d4708b99c3708d617064266932d1a3d139a252ea22f45098fee144c98
repository bf16cbdf opened (version 2, compression level auto)                                                                                                       Unable to load tree ee3de29ac01a25ac924824afb3d9fd7108b7c348f489ef4da68c1491e6b00dad                                                                                  ... which belongs to snapshot 242177ece4c27af96e8414ca8949d715494e663a4d1a77c6927d2a528644185e                                                                      Unable to load tree e64c452bf1c3ebd62793eb4626d8344498a421b6dfff652f184a3fc164d91fc5                                                                                  ... which belongs to snapshot 242177ece4c27af96e8414ca8949d715494e663a4d1a77c6927d2a528644185e
Unable to load tree e64c452bf1c3ebd62793eb4626d8344498a421b6dfff652f184a3fc164d91fc5
 ... which belongs to snapshot 242177ece4c27af96e8414ca8949d715494e663a4d1a77c6927d2a528644185e
Unable to load tree e64c452bf1c3ebd62793eb4626d8344498a421b6dfff652f184a3fc164d91fc5
 ... which belongs to snapshot 242177ece4c27af96e8414ca8949d715494e663a4d1a77c6927d2a528644185e
Unable to load tree 7c28259b1eb78ec17fcf8a861bf574a8eaa1ff19f40747bd5927fe45ed8b609f                                                                                  ... which belongs to snapshot 242177ece4c27af96e8414ca8949d715494e663a4d1a77c6927d2a528644185e
Unable to load tree 01b02a3b8e5e398fe575ade2ce3a0836a4017db4a49eddf6c10fcf197de559f5
 ... which belongs to snapshot 57f4f8a3d74c8bee8927774dbbcc535342ad318ec9778e041dbda2cbaf363632
Unable to load tree 5b6f504f118730a998897f7619e1e0cd432f653833d94949193e7921b01f5acb
 ... which belongs to snapshot 9de40b1cadafd5d4c7f966e743df17feaa208a0492039552e56cd2ef221f2948
Unable to load tree 41ebc38d0b194067d1b2060bca47309e31a0f3ff2950b5954b2c6c732d2701da                                                                                  ... which belongs to snapshot fddd580112dce630063f7dacc0e2655284c96b01f7bf69e1673e3c735f52a939
Unable to load tree 546cf37872c04a9572d1c61ac19d8971795d54ac3893f1491b9465e680b86551
 ... which belongs to snapshot 2405300288695ecd8ed4fa6edfd3c7787c2042226c619cfe4f24882c1b1d96eb
Unable to load tree dcd3d041eb398f4f889dc9d092d8bdb51250e8c482efb4d5c01661517e4efed5
 ... which belongs to snapshot 29a2f852bc492b40ccc92567ed0198cf200c9a2700650747bcf311643b6d1f78
Unable to load tree 01538af67eb9d4ea31fbd7221a1d907b33179b541f7ec458b4f3f6bcdb758f4b                                                                                  ... which belongs to snapshot 51cb7601ee5d0aeac6d782b778dcfec5ba0d2f4da71dfd460f3ce5e2c908c73a                                                                      Unable to load tree 743e592d698f16d3a8459cf82a18938ba485611a9f804a9d3f1c0e2da2cdcb2c                                                                                  ... which belongs to snapshot 26865cdf09011c0c514efa6a65d2e4c3dc4e3884bbea7323f49fd7a31d5d2c3b                                                                      Unable to load tree 4f34dce38488497792580c4ea89c9ae111fb4f64828664f18de24173e9fc6caf                                                                                  ... which belongs to snapshot 450e5d3aaec5257c088480c1bd5056426fe0cce2ade4ebddcc47b51cfba0a74f                                                                      Unable to load tree 67d905f077124b7958711ffd21c02deca4f9bd8435a7afbeca0a541b3f51f22c                                                                                  ... which belongs to snapshot 17e6dd4583786f08d9a53012a9887e8c8a388304f2afef7895e18fb82ca069ed
Unable to load tree 72f239a19efc711c76c467eff5f8e3db26a18cf75955d0d4e450fa7bc97eaa0f                                                                                  ... which belongs to snapshot be1cf5b0a5323d8cfa319553898e40cff3e006a77290751d22259151e49fe223                                                                      Unable to load tree 425620b46186fec6bbd825ea82a3c43288de5e933c1dfe45129ec872cc342363                                                                                  ... which belongs to snapshot b5522192a7f48cf50829336f7f43db026f016b9f073a418e1029e523695ae169                                                                      Unable to load tree ed7f65c58c19f33b4c362198e6a1fd4b0d4055c63e621733384050aa780aa0d4                                                                                  ... which belongs to snapshot a7d62a9b09c15aa6c072ca7aefc02a862eb53c0dcd2af8a95039ca7035e39665                                                                      Unable to load tree 2d5bd789e979eeba0617f32ff0969035a2fb63b7df86845909d1bee9117948c7                                                                                  ... which belongs to snapshot 892c1a557271ececa7fb67d8f75766e770f97a97aa4e438b044017461d67f11b                                                                      Unable to load tree 41866e8b3f79b4f1c4c65ce1a34f26e55d906f6ecf4806d3aaed32849aaee554                                                                                  ... which belongs to snapshot 1499545d33c070c0b80995dc3fe48672d2001f3f253ceb9e7bf6b94f170dcdf6                                                                      Unable to load tree e828da834beec211d85bc740ad9e0ad2ae4dc39a6143332c2e3a8a091767cb86                                                                                  ... which belongs to snapshot d359f8d2e95a0d108dc7386996a0e8454216b220ccc3fa98971d96214764c9ab                                                                      Unable to load tree 111331ce08a2eba2c08675f60509a90ec8a293612fcfdc3f5bcb650f8ae9e15b                                                                                  ... which belongs to snapshot 22aa1cf7b3f7fc0a55b0bd95cda2f342236a381a95f5e14b0d02967d79ec02f3                                                                      Unable to load tree 4143bc531f630c4ed5f5790de67d6e41c4d244150a4e1a2dba72ff2e7038336c                                                                                  ... which belongs to snapshot 17dbfd0209e0399fd84973775041d6a448db2b0699239f79726c12c8959097d3
2

Single disk will always result with errors - it is not if but when. You can check your disk specification - probably something like 1 error in 10^15 bits read.

restic check just have means to identify it thanks to checksums and ability to verify every file validity. For example any relocated sector - which disk can mange - results in stored data corruption. Few sectors will be re-mapped to so spare area. But whatever they contained is gone.

Only solution is to use filesystem and setup which have ability to mitigate it - e.g. mirrored BTRFS or ZFS.

4

I did check the disk. badblocks did not find any.

5

What happens when disk detects bad sector is that it is re-allocated. With all physical sector replaced by new empty one. Then badblock does not detect anything indeed. But data is corrupted. Check smart data and look at:

5 Reallocated_Sector_Ct   0x0033   100   100   050    Pre-fail  Always       -       8
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       1

Above single recorded reallocation event. I was able to identify file affected - restore it from backup and all is working. Badblocks does not show any errors neither.

Of course it is only theory what happened here. But I have seen it myself many times - mostly on mechanical disks. Usually it means that just one file gets corrupted and disk can work without any extra error for many years after.

Could be RAM error etc.

The best advice is to have multiple backups. One gets corrupted - just delete or sync from another backup and move on.

With restic you can try to repair your repo - sometimes it works - Troubleshooting — restic 0.16.4 documentation

6

OK, I’t may be a hardware error that’s causing it. But how do I proceed to salvage the repository?

7

I have added link in my previous post how to try to salvage it.

But most reliable way is just start fresh - sync from another backup. This is what I do. Repairing repo is really the last resort.

8 
root@pi01:~# smartctl -a /dev/sda1
smartctl 7.2 2020-12-30 r5155 [aarch64-linux-6.1.21-v8+] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               WD
Product:              My Passport 0810
Revision:             1049
Compliance:           SPC-4
User Capacity:        1,000,170,586,112 bytes [1.00 TB]
Logical block size:   512 bytes
Physical block size:  4096 bytes
Rotation Rate:        5400 rpm
Serial number:        WXC1AA3R2169
Device type:          disk
Local Time is:        Sun Mar 24 11:30:35 2024 GMT
SMART support is:     Unavailable - device lacks SMART capability.

=== START OF READ SMART DATA SECTION ===
Current Drive Temperature:     0 C
Drive Trip Temperature:        0 C

Error Counter logging not supported

No Self-tests have been logged

and


root@pi01:~# dumpe2fs -b /dev/sda1
dumpe2fs 1.46.2 (28-Feb-2021)
root@pi01:~#

No blocks reserved as bad in the filesystem.

9

No. Your USB enclosure does not pass SMART data. Only basic INFO.

10

Bummer. Is there at definitive way to check the disk? Neither badblocks nor the filesystem reports anything bad.

11

It means that your disk is OK now. But does not exclude bad sectors which were reallocated in the past. badblocks would only detect bad sector if luckily reads it first - before any other program.

Only way is to use at least mirror and filesystem capable of self healing - BTRFS works perfectly fine or raspberry pi.