Ciphertext verification failed during restore, `restic check` found no errors

[fd0]

@rawtaz the ID 2857a160da9758723af784bc7fb5a90d48b553872f68246e4e38fff3409813c7 identifies the blob within the pack file, of which the file name is 9b5a3cf4ccea5e20557c62e6e9eef27cf529ac13a42d65b7e10592d847976042

Finding the pack for a blob is possible by looking at the index. You can list the index files via restic list index and then decrypt and print them to stdout via restic cat index ID, which can in turn be fed to jq for pretting printing:

$ restic cat index f94c98288155eca29801a585946e14d16e8f8e9a520346b0d5334e2263f120e5 | jq .
[...]
    {
      "id": "6ad6006670f01718e6b2ae6c9b4e1b3dc78c949e92531dcc7a75434b53d3e41d",
      "blobs": [
        {
          "id": "b3928295a547b90bd8ba92d84e3590d181a19fc4be5f1e7e576487ca56103864",
          "type": "data",
          "offset": 0,
          "length": 846437
        },
        {
          "id": "78d02070292e97dda5f65b175ece6a120a5d375d86917dd5a2aa789677a71e43",
          "type": "data",
          "offset": 2667780,
          "length": 604370
        },
        {
          "id": "69b24d1bda844f41710e45976904f6425ed0bfb57b7e8b736e9941261d879549",
          "type": "data",
          "offset": 3272150,
          "length": 927341
        },
        {
          "id": "13d54ce2efb071b64f3bbcabbec240ac6fb3b54ef52fda3d6e83898424395bcd",
          "type": "data",
          "offset": 846437,
          "length": 1821343
        }
      ]
    },
[...]

There you can see four blobs contained in a pack file. The same contents (roughly) is also contained in the header at the very end of the pack file.

@Sitwon

I still am not clear on how (if it is at all possible) to repair this without having to start over with a new repository. I am hoping it is possible, because if this repository contained several months or years of snapshots I would not want to throw away all of that history. Even if this was caused by a silent hard drive corruption (which I find unlikely in this scenario), how would you salvage the existing good data and unaffected snapshots?

To be honest, restic is not handling this situation very well. A solution would be:

• Save all blobs in the same pack file which are unmodified to individual files
• Remove the pack file
• Run restic rebuild-index, so that the index is newly built from the available pack files and doesn’t contain any references to the broken blob any more
• Then backup the saved blobs in the files from the first step. That will insert them into the repo, most likely as individual blobs.

Afterwards, there’s only the one single blob missing from the repo. If this blob is found somewhere during the next backup, it is saved again in the repo and everything is repaired. If it isn’t found, then all files referencing that blob cannot be fully restored.

We have some plans to address this situation by adding recovery information to the pack files, but these are only rough plans right now. You can find the discussion here: Feature: Add redundancy for data stored in the repository · Issue #256 · restic/restic · GitHub

@Sitwon

To prevent corruption if the backup is interrupted, the safer approach is to write to a temp file, then move the tempfile to the target path. Because if you start writing to data/xx/xxxx… and the connection gets interrupted you’ll end up with a partially-written file. But moves are an atomic file system operation, so once the data is on the file system, it will either move and be complete, or it won’t move. There’s no risk of a partial file.

You’re right, we’re directly writing the data in the local and sftp backends. We used to write to a tempfile and then move it around, but it turned out that this is not ideal for some file systems. Moving is not guaranteed to be an atomic operation, that depends on the file system. For several popular fuse-based file systems it’s not the case, for example.

What we’re trying instead is to tell the fs via fsync() to flush the remaining data to disc, and only at the very end add the contents of the newly written pack file to the index file. When anything happens and the process is interrupted, the pack file hasn’t been added to the index yet and when you start restic again, the data is uploaded again. The remainders are cleaned up by the next run of restic prune.

The prune operation (and rebuild-index) will read the header at the end of the pack file. If that is intact, the file has been written completely and can be used. Otherwise it is removed as a leftover from an interrupted operation.

In practice (even with terabytes of data) this turned out to work quite well.