Fatal: config cannot be loaded: ciphertext verification failed

I’m getting the error in the subject line whenever I try to perform any actions on one of my repos. The only relevant link I was able to find was https://github.com/restic/restic/issues/2725 and the solution for that seemed inapplicable to me.

$ restic version
restic 0.9.6 compiled with go1.13.6 on linux/amd64

restic snapshots --repo "gs:<google stuff here>:/<my repo>"  --last
Fatal: Fatal: config cannot be loaded: ciphertext verification failed

Backend: Google

My understanding from the issue linked above is that this is caused by restic having a corrupted binary config in the repo. Since I don’t have a synced version like the guy above did, I’d like to know if there’s a way I can recover/regenerate that config somehow.

Thoughts?

Please take a short look at the config file: Is it empty, can you access it manually? Is its size exactly 155 bytes? If the file itself looks like it might be ok (obviously there’s not telling whether these random binary characters are valid by looking at them), then the next question is how it got corrupted. Restic only every writes the config file when creating a new repository. Please make sure you know why the file got damaged in the first place, before trying the following command:

I’ve put together a small command which will create a new “fake” config file for a repository which no longer has one. The “fake” part means that the repository gets a new random id and new random chunker parameters, which will break deduplication but are enough to allow you to read the data in the repository.

To use it checkout the change-repo-id branch at https://github.com/MichaelEischer/restic/tree/change-repo-id and compile restic using go build -tags debug ./cmd/restic (feel free to ask if you need more detailed steps). Then move the broken config file in the repository to somewhere else (but make sure to keep it for now!). Afterwards ./restic debug fakeConfig will add a new “fake” config file to the repository. I’d recommend to run ./restic check --read-data (which will read all data in the repository) to let restic verify the integrity of everything in that repository.
Note: the check command of the custom built restic binary includes several optimizations from the master branch which speed-up the check command a lot.

It’s exactly 155 bytes, as you suggested. The contents are indeed binary.

Just to make things really interesting, in my copy of the local cache, the most recent updates are say 25th’s August, whereas the config file on Google Cloud claims it hasn’t been updated since the 14th of August (which is a plausible date for the creation of this repo). All my other repos are working fine though.

Regarding the change you’ve put together, I’d like to check one thing: will this mean that deduplication will work in future for this repo, and the old ones will remain deduplicated, but if there are snapshots on the old and new setups, they won’t share data?

I tested your solution. After the build, I did:

./restic debug fakeConfig
./restic check --read-data

The debug command command said “debug enabled”, and then gave me the standard usage info.
The restic check gave me the same error as before (about ciphertext).
The timestamp on the config file did not change.

Was I supposed to run restic debug changeID at some point?

Output from fakeConfig command:

# ./restic debug fakeConfig --repo <repo name>
debug enabled
Debug commands

Usage:
  restic debug [command]

Available Commands:
  dump        Dump data structures

Flags:
  -h, --help   help for debug

Global Flags:
      --block-profile dir             write block profile to dir
      --cacert file                   file to load root certificates from (default: use system certificates)
      --cache-dir directory           set the cache directory. (default: use system default cache directory)
      --cleanup-cache                 auto remove old cache directories
      --cpu-profile dir               write cpu profile to dir
      --insecure-kdf                  use insecure KDF settings
      --json                          set output mode to JSON for commands that support it
      --key-hint key                  key ID of key to try decrypting first (default: $RESTIC_KEY_HINT)
      --limit-download int            limits downloads to a maximum rate in KiB/s. (default: unlimited)
      --limit-upload int              limits uploads to a maximum rate in KiB/s. (default: unlimited)
      --listen-profile address:port   listen on this address:port for memory profiling
      --mem-profile dir               write memory profile to dir
      --no-cache                      do not use a local cache
      --no-lock                       do not lock the repo, this allows some operations on read-only repos
  -o, --option key=value              set extended option (key=value, can be specified multiple times)
      --password-command command      specify a shell command to obtain a password (default: $RESTIC_PASSWORD_COMMAND)
  -p, --password-file file            read the repository password from a file (default: $RESTIC_PASSWORD_FILE)
  -q, --quiet                         do not output comprehensive progress report
  -r, --repo repository               repository to backup to or restore from (default: $RESTIC_REPOSITORY)
      --tls-client-cert file          path to a file containing PEM encoded TLS client certificate and private key
      --trace-profile dir             write trace to dir
  -v, --verbose n                     be verbose (specify --verbose multiple times or level --verbose=n)

Use "restic debug [command] --help" for more information about a command.

Yep, snapshots created after writing a new config file will not deduplicate properly with older snapshots. However, deduplication will work between new snapshots. This deduplication deficit only applies to files >0.5 MB and only if they’ve change since their parent snapshot.

The help output looks like you’ve built the master branch and not the change-repo-id branch. You shouldn’t to run restic debug changeID (well, actually with a broken config file, there’s no chance to convince that command to do anything). The fakeConfig command will refuse to make any changes if it can find anything which might be a config file. That is you have to move the config file somewhere else and afterwards run ./restic debug fakeConfig.

# ./restic debug fakeConfig --repo "gs:<cloud>:/<directory>"
debug enabled
Load(<config/0000000000>, 0, 0) returned error, retrying after 462.318748ms: googleapi: got HTTP response code 404 with body: No such object: <cloud>/<directory>/config

You’re right, I had built the wrong branch; sorry. I got the above output from restic debug fakeConfig. Note that I got the same error message (the line beginning “Load”) 10 times, just with a different number of ms in the retry time.

I ran the other command anyway, and got lots of:

error: error loading index d9dca5dc: ciphertext verification failed

Thanks! HTH,

Oh, I forgot to mention that this warning is expected to be printed exactly ten times. I was too lazy to remove them for the debug command.

The new “ciphertext verification failed” errors look bad. Can you check whether the key files are intact? (If I’m not completely mistaken, then restic isn’t even able to load damaged key files. But let’s just make sure that these are not the problem) Just run shasum -a256 <key-files> on the files in the keys directory of the repository. The filename must match the calculated SHA-256 hash for each key file.

Any thoughts as to how I do that with a Google Cloud storage?

restic list keys and restic cat key <keyid> should work to download the key. Then just pipe the result through shasum. Or do you have some kind of web interface to access the keys folder of the repository?

I got the following results:

$ restic cat --repo "gs:CLOUD:/REPO" "key" "ea7edf1e223e5e62c80c1ad550d59aa5b78aa09d98457d7e90ece60dd4f99001" --quiet
{
  "created": "2020-08-14T09:07:13.768580236+08:00",
  "username": "USER",
  "hostname": "HOST",
  "kdf": "scrypt",
  "N": 32768,
  "r": 8,
  "p": 3,
  "salt": "SALT",
  "data": "DATA"
}
$ restic cat --repo "gs:CLOUD:/REPO" "key" "ea7edf1e223e5e62c80c1ad550d59aa5b78aa09d98457d7e90ece60dd4f99001" --quiet | shasum -a256
70cf4bed4212f77fd421c5f70ee44f848681de215f3bb3aa7888a609382f41e9  -
$ restic cat --repo "gs:CLOUD:/REPO" "key" "661b987e9a04bdeda4da87a908fb23d56d63ee3975b10c1500b99b5229e874e5" --quiet
{
  "created": "2020-08-25T08:32:14.566238264+08:00",
  "username": "USER",
  "hostname": "HOST",
  "kdf": "scrypt",
  "N": 32768,
  "r": 8,
  "p": 3,
  "salt": "SALT",
  "data": "DATA"
}
$ restic cat --repo "gs:CLOUD:/REPO" "key" "661b987e9a04bdeda4da87a908fb23d56d63ee3975b10c1500b99b5229e874e5" --quiet | shasum -a256
c15a2243cb58e5ff05c42c81fdf2ba2309f31058b4b2ba2de1e3a793df0899ec  -

As you can see, it’s successfully fetched the key from the repo, but it’s not generating the same checksum. I tried the --quiet option because it seemed like it was relevant. Is it possible that restic has reformatted the key, and I need to give “cat” some kind of option?

You’re right about the reformatting. I forgot that the cat command pretty prints some of the internal objects, sorry. Well, the cat command in the change-repo-id branch has as of now a --raw option, which works on my test repository.

That works; thanks!

# restic cat --repo "gs:CLOUD:/REPO" "key" "ea7edf1e223e5e62c80c1ad550d59aa5b78aa09d98457d7e90ece60dd4f99001" --raw --quiet | shasum -a256
debug enabled
ea7edf1e223e5e62c80c1ad550d59aa5b78aa09d98457d7e90ece60dd4f99001  -
# restic cat --repo "gs:CLOUD:/REPO" "key" "661b987e9a04bdeda4da87a908fb23d56d63ee3975b10c1500b99b5229e874e5" --raw --quiet | shasum -a256
debug enabled
661b987e9a04bdeda4da87a908fb23d56d63ee3975b10c1500b99b5229e874e5  -

So, we can see that the numbers match up. Do you have any further recommendations?

Thanks again!

I think the next step is to rebuild the repository index as it’s apparently damaged. I’d recommend to create a backup of the index folder first (you might want to give rclone a try). Then run restic rebuild-index afterwards.

I’ve tried that now. I got lots of errors like:

pack file cannot be listed fff0972aab91e5a208f58671707e7d64b92f9e9d93bf4bd9e52532d141de6fcc: ciphertext verification failed

Even after doing that, when I run restic snapshots, I get lots of this:

could not load snapshot 001b01a0: ciphertext verification failed

Hmm, that starts to look like the key does not match the repository. How comes that your repository contains two keys? Please try restic cat <snapshotID> together with the --key-hint keyID flag for both keys.

First run:

restic cat --repo "gs:CLOUD:/PATH" "snapshot" "fff7bd4a" "--key-hint" "661b987e9a04bdeda4da87a908fb23d56d63ee3975b10c1500b99b5229e874e5"
repository 44e61d8e opened successfully, password is correct
ciphertext verification failed
github.com/restic/restic/internal/crypto.init
    internal/crypto/crypto.go:30
runtime.doInit
    /usr/lib/golang/src/runtime/proc.go:5222
runtime.doInit
    /usr/lib/golang/src/runtime/proc.go:5217
runtime.doInit
    /usr/lib/golang/src/runtime/proc.go:5217
runtime.doInit
    /usr/lib/golang/src/runtime/proc.go:5217
runtime.main
    /usr/lib/golang/src/runtime/proc.go:190
runtime.goexit
    /usr/lib/golang/src/runtime/asm_amd64.s:1357

Second run:

restic cat --repo "gs:CLOUD:/PATH" "snapshot" "fff7bd4a" "--key-hint" "ea7edf1e223e5e62c80c1ad550d59aa5b78aa09d98457d7e90ece60dd4f99001"
Fatal: Fatal: config cannot be loaded: ciphertext verification failed

I assumed that the second key was created in the fakeConfig thing you had me run, but could be wrong here. Can I delete one of the keys? If so, which? Is there a way I can delete and recreate the key, or does that not work?