Tree could not be loaded

Hello,

I’ve been using restic with Storj (S3 gateway) for around 3 weeks. Unfortunately, I just started to encounter the following error when running a backup:

error: tree 7611d8e7d434deb84957c59af40c13f6fdde990ef35d89bce6b679f4fd813b4d could not be loaded; the repository could be damaged: decrypting blob 7611d8e7d434deb84957c59af40c13f6fdde990ef35d89bce6b679f4fd813b4d failed: ciphertext verification failed

Would someone be able to provide me with some guidance on how to recover from this?

Some info:

  • 2.4 GB of source files
  • I run backup, forget (with --prune and some keep options), and then check (with --read-data-subset=1%) once per day.
  • This feels unrelated, but I am also now seeing Warning: at least one source file could not be read in the backup logs. Prior to this, there were no other warnings or errors in the logs other than those mentioned above.

Also, the backup command exits with code 3 (perhaps due to the warning mentioned above). However, should the earlier error call for a more critical exit code to be returned?

Here are the systemd log entries:

Backing up to storj
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
^M  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0^M  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0^M100     2  100     2    0     0      2      0  0:00:01 --:--:--  0:00:01     2
OK
using parent snapshot ffa8b536
error: tree 7611d8e7d434deb84957c59af40c13f6fdde990ef35d89bce6b679f4fd813b4d could not be loaded; the repository could be damaged: decrypting blob 7611d8e7d434deb84957c59af40c13f6fdde990ef35d89bce6b679f4fd813b4d failed: ciphertext verification failed
Files:         123 new,     3 changed, 29713 unmodified
Dirs:            0 new,     6 changed,  6569 unmodified
Added to the repo: 43.949 KiB
processed 29839 files, 2.373 GiB in 0:10
snapshot ea9e0eb7 saved
Warning: at least one source file could not be read
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
^M  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0^M100     2  100     2    0     0      3      0 --:--:-- --:--:-- --:--:--     3
OK

This is with restic 0.12.1 compiled with go1.17.3 on linux/amd64

Thank you.

restic check --read-data reported no errors:

using temporary cache in /tmp/restic-check-cache-6288288
repository ea994a94 opened successfully, password is correct
created new cache in /tmp/restic-check-cache-6288288
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:15] 100.00%  25 / 25 snapshots
read all data
[6:56] 100.00%  613 / 613 packs
no errors were found

Here’s the result of restic find --show-pack-id --tree 7611d8e7d434deb84957c59af40c13f6fdde990ef35d89bce6b679f4fd813b4d

$ restic find --show-pack-id --tree 7611d8e7d434deb84957c59af40c13f6fdde990ef35d89bce6b679f4fd813b4d
repository ea994a94 opened successfully, password is correct
Unable to load tree 18049df05befbc3cda78a77b161a3d86a44cde2ff1d2e7c44e060b89aa477967
 ... which belongs to snapshot b4a627ef7663261a97cbd9a4e4a0777bdcf0f035e029d1aa823cc4a2880832ac.
Unable to load tree 18049df05befbc3cda78a77b161a3d86a44cde2ff1d2e7c44e060b89aa477967
 ... which belongs to snapshot 4b9cffec31187c22903d15217d2223883298aafd0b0b9890bc610fc1f4c5a0fa.
Unable to load tree 18049df05befbc3cda78a77b161a3d86a44cde2ff1d2e7c44e060b89aa477967
 ... which belongs to snapshot 6d45b871cef1766819ac4bb5af5d9925124560aca1f74e56772f4c581c8d5c33.

<lots of the above>

Object belongs to pack c2c28885ae757bfbb7e3f515db7756189d57edb1da3a2da873e30e15b1063e62
 ... Pack c2c28885: <Blob (tree) 7611d8e7, offset 446847, length 44486>

It appears that one option is to remove the file c2c28885ae757bfbb7e3f515db7756189d57edb1da3a2da873e30e15b1063e62 from the destination storage and then run rebuild-index folllowed by check --read-data. Is there anything I should try before doing so?

I am able to run cat on the above pack file. The following runs without error:

restic cat pack c2c28885ae757bfbb7e3f515db7756189d57edb1da3a2da873e30e15b1063e62

As check seems to work, but the backup command doesn’t, I’d suspect that the cache is broken somehow. You could move restic’s cache folder to somewhere else which might help.

Removing the file from the destination storage will probably cause a lot of other problems.

1 Like

Thanks @MichaelEischer, after moving the cache folder everything appears to be working fine. Is there something analogous to check that can verify the consistency of the cache?

You can run shasum -a256 **/* (or so) on the cache folder. All files except the config should have their sha256 hash as filename. Or you could give the code from GitHub - fd0/psha a try.

1 Like