Help with repository errors

A regularly scheduled run of restic check --read-data-subset=50M resulted in the following error:

error for tree 954a6090:
  decrypting blob 954a609071855f38c2834443a5e94e6892ff00cfba4d89fdb0fc55f8082147ef failed: ciphertext verification failed

Running restic find --tree 954a6090 yielded

repository 91dd5922 opened successfully, password is correct
Found tree 954a609071855f38c2834443a5e94e6892ff00cfba4d89fdb0fc55f8082147ef
 ... path <path>
 ... in snapshot 2f121bbc (2022-05-06 16:00:35)

I then ran restic check --read-data, which resulted in:

using temporary cache in /tmp/restic-check-cache-204863183
repository 91dd5922 opened successfully, password is correct
created new cache in /tmp/restic-check-cache-204863183
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:30] 100.00%  149 / 149 snapshots...
read all data
pack 0c67dba6 contains 1 errors: [blob 5: ciphertext verification failed]
pack 0360798d contains 1 errors: [blob 5: ciphertext verification failed]
[39:21] 100.00%  2237 / 2237 packs...
Fatal: repository contains errors

Running restic find --pack 0c67dba6 resulted in a lot of output of the form

Found blob <blob>
 ... in file <path>
     (tree <tree>)
 ... in snapshot <snapshot> (<timestamp>)

I’m not sure how to proceed. Any help would be greatly appreciated.

This is using restic 0.13.1 compiled with go1.18.1 on linux/amd64.

Running the following seems to indicate a problem with the hash

$ restic cat pack 0c67dba6
repository 91dd5922 opened successfully, password is correct
Fatal: unable to parse ID: invalid length for hash

If I run sha256sum on the pack file, the hash is consistent with the filename

$ sha256sum 0c67dba620b4e9c8a2eb3d5767fe3c75ef49643abec604dfb25eb55119bd7372 
0c67dba620b4e9c8a2eb3d5767fe3c75ef49643abec604dfb25eb55119bd7372  0c67dba620b4e9c8a2eb3d5767fe3c75ef49643abec604dfb25eb55119bd7372

These errors usually indicate bitflips in hardware at the client that created the snapshot. Please run a memcheck and ideally some CPU-stress test like prime95 to make sure the hardware is fine.

Other than that you can follow the steps at Recover from broken pack file · Issue #828 · restic/restic · GitHub to repair your repository.

Thanks @MichaelEischer. It looks like bad RAM was the culprit!

After some more testing I’m now thinking that my RAM is fine, however a bitflip might have occurred for some other reason. Regardless, I’m trying to fix my repository and am confused by the following results.

I cleared the restic cached, and running restic check --read-data now yields no errors:

repository 91dd5922 opened successfully, password is correct
created new cache in /tmp/restic-check-cache-3392818683
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:13] 100.00%  161 / 161 snapshots...
read all data[12:44] 100.00%  2229 / 2229 packs...
no errors were found

However, running restic cat pack 0c67dba6 still reveals problems:

created new cache in /home/ken/.cache/restic
Fatal: unable to parse ID: invalid length for hash

What does this mean, and how can I fix my repository?

Thanks.

That error only means that you called cat pack ... incorrectly. The command expects a full pack id (64 hex characters), not the shortened variant.

Oh my, that’s embarrassing! I’m going to assume that since check --read-data passes now, my repository should be fine. Thanks again for your help.