I’m getting the error in the subject line whenever I try to perform any actions on one of my repos. The only relevant link I was able to find was https://github.com/restic/restic/issues/2725 and the solution for that seemed inapplicable to me.
$ restic version
restic 0.9.6 compiled with go1.13.6 on linux/amd64
restic snapshots --repo "gs:<google stuff here>:/<my repo>" --last
Fatal: Fatal: config cannot be loaded: ciphertext verification failed
Backend: Google
My understanding from the issue linked above is that this is caused by restic having a corrupted binary config in the repo. Since I don’t have a synced version like the guy above did, I’d like to know if there’s a way I can recover/regenerate that config somehow.
Please take a short look at the config file: Is it empty, can you access it manually? Is its size exactly 155 bytes? If the file itself looks like it might be ok (obviously there’s not telling whether these random binary characters are valid by looking at them), then the next question is how it got corrupted. Restic only every writes the config file when creating a new repository. Please make sure you know why the file got damaged in the first place, before trying the following command:
I’ve put together a small command which will create a new “fake” config file for a repository which no longer has one. The “fake” part means that the repository gets a new random id and new random chunker parameters, which will break deduplication but are enough to allow you to read the data in the repository.
To use it checkout the change-repo-id branch at https://github.com/MichaelEischer/restic/tree/change-repo-id and compile restic using go build -tags debug ./cmd/restic (feel free to ask if you need more detailed steps). Then move the broken config file in the repository to somewhere else (but make sure to keep it for now!). Afterwards ./restic debug fakeConfig will add a new “fake” config file to the repository. I’d recommend to run ./restic check --read-data (which will read all data in the repository) to let restic verify the integrity of everything in that repository.
Note: the check command of the custom built restic binary includes several optimizations from the master branch which speed-up the check command a lot.
It’s exactly 155 bytes, as you suggested. The contents are indeed binary.
Just to make things really interesting, in my copy of the local cache, the most recent updates are say 25th’s August, whereas the config file on Google Cloud claims it hasn’t been updated since the 14th of August (which is a plausible date for the creation of this repo). All my other repos are working fine though.
Regarding the change you’ve put together, I’d like to check one thing: will this mean that deduplication will work in future for this repo, and the old ones will remain deduplicated, but if there are snapshots on the old and new setups, they won’t share data?
The debug command command said “debug enabled”, and then gave me the standard usage info.
The restic check gave me the same error as before (about ciphertext).
The timestamp on the config file did not change.
Was I supposed to run restic debug changeID at some point?
Output from fakeConfig command:
# ./restic debug fakeConfig --repo <repo name>
debug enabled
Debug commands
Usage:
restic debug [command]
Available Commands:
dump Dump data structures
Flags:
-h, --help help for debug
Global Flags:
--block-profile dir write block profile to dir
--cacert file file to load root certificates from (default: use system certificates)
--cache-dir directory set the cache directory. (default: use system default cache directory)
--cleanup-cache auto remove old cache directories
--cpu-profile dir write cpu profile to dir
--insecure-kdf use insecure KDF settings
--json set output mode to JSON for commands that support it
--key-hint key key ID of key to try decrypting first (default: $RESTIC_KEY_HINT)
--limit-download int limits downloads to a maximum rate in KiB/s. (default: unlimited)
--limit-upload int limits uploads to a maximum rate in KiB/s. (default: unlimited)
--listen-profile address:port listen on this address:port for memory profiling
--mem-profile dir write memory profile to dir
--no-cache do not use a local cache
--no-lock do not lock the repo, this allows some operations on read-only repos
-o, --option key=value set extended option (key=value, can be specified multiple times)
--password-command command specify a shell command to obtain a password (default: $RESTIC_PASSWORD_COMMAND)
-p, --password-file file read the repository password from a file (default: $RESTIC_PASSWORD_FILE)
-q, --quiet do not output comprehensive progress report
-r, --repo repository repository to backup to or restore from (default: $RESTIC_REPOSITORY)
--tls-client-cert file path to a file containing PEM encoded TLS client certificate and private key
--trace-profile dir write trace to dir
-v, --verbose n be verbose (specify --verbose multiple times or level --verbose=n)
Use "restic debug [command] --help" for more information about a command.
Yep, snapshots created after writing a new config file will not deduplicate properly with older snapshots. However, deduplication will work between new snapshots. This deduplication deficit only applies to files >0.5 MB and only if they’ve change since their parent snapshot.
The help output looks like you’ve built the master branch and not the change-repo-id branch. You shouldn’t to run restic debug changeID (well, actually with a broken config file, there’s no chance to convince that command to do anything). The fakeConfig command will refuse to make any changes if it can find anything which might be a config file. That is you have to move the config file somewhere else and afterwards run ./restic debug fakeConfig.
# ./restic debug fakeConfig --repo "gs:<cloud>:/<directory>"
debug enabled
Load(<config/0000000000>, 0, 0) returned error, retrying after 462.318748ms: googleapi: got HTTP response code 404 with body: No such object: <cloud>/<directory>/config
You’re right, I had built the wrong branch; sorry. I got the above output from restic debug fakeConfig. Note that I got the same error message (the line beginning “Load”) 10 times, just with a different number of ms in the retry time.
I ran the other command anyway, and got lots of:
error: error loading index d9dca5dc: ciphertext verification failed
Oh, I forgot to mention that this warning is expected to be printed exactly ten times. I was too lazy to remove them for the debug command.
The new “ciphertext verification failed” errors look bad. Can you check whether the key files are intact? (If I’m not completely mistaken, then restic isn’t even able to load damaged key files. But let’s just make sure that these are not the problem) Just run shasum -a256 <key-files> on the files in the keys directory of the repository. The filename must match the calculated SHA-256 hash for each key file.
restic list keys and restic cat key <keyid> should work to download the key. Then just pipe the result through shasum. Or do you have some kind of web interface to access the keys folder of the repository?
As you can see, it’s successfully fetched the key from the repo, but it’s not generating the same checksum. I tried the --quiet option because it seemed like it was relevant. Is it possible that restic has reformatted the key, and I need to give “cat” some kind of option?
You’re right about the reformatting. I forgot that the cat command pretty prints some of the internal objects, sorry. Well, the cat command in the change-repo-id branch has as of now a --raw option, which works on my test repository.
I think the next step is to rebuild the repository index as it’s apparently damaged. I’d recommend to create a backup of the index folder first (you might want to give rclone a try). Then run restic rebuild-index afterwards.
Hmm, that starts to look like the key does not match the repository. How comes that your repository contains two keys? Please try restic cat <snapshotID> together with the --key-hint keyID flag for both keys.
I assumed that the second key was created in the fakeConfig thing you had me run, but could be wrong here. Can I delete one of the keys? If so, which? Is there a way I can delete and recreate the key, or does that not work?
Sorry for the late reply. This test confirm that your repository has two key files which contain a different masterkey. 661b987e is the one which was used when creating the fakeConfig file. ea7edf1e2 is with some luck the correct key for the repository. Please restore the original config and check whether you can load the snapshot using the second key. (or delete the current config and create a new one using fakeConfig)
I assumed that the second key was created in the fakeConfig thing you had me run, but could be wrong here. Can I delete one of the keys? If so, which? Is there a way I can delete and recreate the key, or does that not work?
The fakeConfig code does not modify the keys in any way. Recreating a key from scratch is not possible, as that would require breaking the repository encryption (which would take a few million years or more). I wouldn’t delete any keys before we know which one is correct.
Could you compare the output of restic cat masterkey with both key hints and check whether the difference is minimal (one or two characters) or much larger (Please don’t post the raw output of that command, it contains the actual keys used for encryption)?
Hello @MichaelEischer. I don’t want to hijack this thread but this is somewhat related…
Would having two keys cause an error?
I created two different repositories on the same Service such as One Drive. I am backing up two different locations so I wanted them to have different passwords. For example. One Repository is under Restic02/MIS01 and the other was under Restic02/DUP02
Would these be causing an issue? I am getting a lot of these. I have been trying to clean up the repository and I still keep getting errors. I am wondering if having two keys on the same One Drive Location would be causing some of these issues. Sometimes I have to issue the unlock command. I figured it would be a lock previously that was perhaps the culprit…Or it could be the repository being accessed simultaneously?
Thanks.
I’m not really sure I understand your repository setup. If you have two repositories in different paths (e.g. rclone:onedrive:/Restic02/MIS01 and rclone:onedrive:/Restic02/DUP02) then these should be completely independent. Was one of the repositories created as a copy of the other one?