Hello,
we are experiencing several ciphertext verification failed
errors for different repositories and also different hosts.
We are using restic 0.16.4 as a sidecar container (inside Kubernetes) to backup Strapi volumes (consisting of a SQLite3 database and several static content files like images) to S3 object storage.
We now had 3 different cases were we got that special ciphertext verification failed
error. And there seems to be no way to recover from that.
BTW: we also use exactly the same sidecar container for other (much larger) volume backups without any problems. Those Strapi volume backups are quite small - like 7-20 MB. Therefore I assume that we are hitting a special case with the Strapi volume somehow.
All 3 cases happened in different Kubernetes clusters and therefore on different VMs (also on different hardware) and in fact at different hosters too. Also the target S3 object storages were from 3 different locations from 2 distinct providers.
Therefore I really assume that hardware failures are rather unlikely for all those cases.
All 3 cases happened in the last 2 or 3 weeks and only on Strapi volumes even though we are backing up many more volumes to S3 using restic.
In one case we just deleted the whole backup repository (using rclone as restic does not have an option to do that). Both other cases are still in failure state and could be used for debug purposes. I also downloaded the repositories from S3 to recheck the repository locally - without any change.
Here are some outputs. I really hope you can help fixing that problem.
restic version
$ restic version
restic 0.16.4 compiled with go1.21.6 on linux/amd6
restic stats # Shows error
$ restic stats
repository b9d21369 opened (version 2, compression level auto)
[0:00] 0.00% 0 / 4 index files loaded
ciphertext verification failed
github.com/restic/restic/internal/crypto.init
/restic/internal/crypto/crypto.go:30
runtime.doInit1
/usr/local/go/src/runtime/proc.go:6740
runtime.doInit
/usr/local/go/src/runtime/proc.go:6707
runtime.main
/usr/local/go/src/runtime/proc.go:249
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1650
restic check --read-data # Shows error
$ restic check --read-data
using temporary cache in /tmp/restic-check-cache-3789657315
repository b9d21369 opened (version 2, compression level auto)
created new cache in /tmp/restic-check-cache-3789657315
create exclusive lock for repository
load indexes
[0:00] 100.00% 4 / 4 index files loaded
error: error loading index a1243cf997162370e57ad2589f605271922748b20f8800b8090cfb1752b52285: ciphertext verification failed
error: error loading index e83ccb66f95576b834669f31fb7f8e69059a4e343eacba679c6941fba66df4a3: ciphertext verification failed
error: error loading index a75ea15927343280ab9a4dfdbf6615f6c1489dc89c132f334f8f866159e0ac1b: ciphertext verification failed
error: error loading index 474bc50b2c62b12fc3ac8d7127c9583a6468f6e652bab263c8b78e14896f553f: ciphertext verification failed
Fatal: LoadIndex returned errors
restic list blobs # Shows error
$ restic list blobs
repository b9d21369 opened (version 2, compression level auto)
ciphertext verification failed
github.com/restic/restic/internal/crypto.init
/restic/internal/crypto/crypto.go:30
runtime.doInit1
/usr/local/go/src/runtime/proc.go:6740
runtime.doInit
/usr/local/go/src/runtime/proc.go:6707
runtime.main
/usr/local/go/src/runtime/proc.go:249
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1650
restic list index
$ restic list index
repository b9d21369 opened (version 2, compression level auto)
e83ccb66f95576b834669f31fb7f8e69059a4e343eacba679c6941fba66df4a3
a1243cf997162370e57ad2589f605271922748b20f8800b8090cfb1752b52285
474bc50b2c62b12fc3ac8d7127c9583a6468f6e652bab263c8b78e14896f553f
a75ea15927343280ab9a4dfdbf6615f6c1489dc89c132f334f8f866159e0ac1b
restic list packs
$ restic list packs
repository b9d21369 opened (version 2, compression level auto)
86b25cef771f189ca565bc19c99ed6c7c9e3691ba3758079179a600c58e06f88
0016e73b5df0434ddb2642ca4b66378b1bc9ba7d903721b023ece405dbb7047a
41607a0f83dd1d390189fe72059f4808dc633ed865b869a1c64a63925161185d
cfd2ce39624aa9a12d3656deecbf5ecc02144fc277b84e0b1e30669527e80d62
ae6b17714a70a19870c998ce0e9a28734f2ea0f471aec6832791dc8ce0ffaa69
ada0c8097a969bf9d126b39d84da8bfbf5dc428a354c31151c3e1cc86e742fce
restic list snapshots
$ restic list snapshots
repository b9d21369 opened (version 2, compression level auto)
61778ecfa0b209c6eb95a9c07a5058cf80bddb28e427a42e7f6495e377b06d61
74c189425790faf2497ddface2a041ba7fbb883223fa8c93235b8d1b3d7f5d2c
104598c8811d078906e5f1d13044e83d826df034cfb3a1c86a31ac7b317d7876
7d9cf8d381ff6651d5761f9fd8d15566d142e70d827380500b47708eaa4840c3
restic snapshots # Shows error
$ restic snapshots
repository b9d21369 opened (version 2, compression level auto)
Ignoring "74c189425790faf2497ddface2a041ba7fbb883223fa8c93235b8d1b3d7f5d2c": failed to load snapshot 74c18942: ciphertext verification failed
Ignoring "104598c8811d078906e5f1d13044e83d826df034cfb3a1c86a31ac7b317d7876": failed to load snapshot 104598c8: ciphertext verification failed
Ignoring "61778ecfa0b209c6eb95a9c07a5058cf80bddb28e427a42e7f6495e377b06d61": failed to load snapshot 61778ecf: ciphertext verification failed
Ignoring "7d9cf8d381ff6651d5761f9fd8d15566d142e70d827380500b47708eaa4840c3": failed to load snapshot 7d9cf8d3: ciphertext verification failed
ls -R of repository
# File PASSWORD contains the RESTIC_PASSWORD - just for debugging purposes
$ ls -R
.:
config data index keys locks PASSWORD snapshots
./data:
00 41 86 ad ae cf
./data/00:
0016e73b5df0434ddb2642ca4b66378b1bc9ba7d903721b023ece405dbb7047a
./data/41:
41607a0f83dd1d390189fe72059f4808dc633ed865b869a1c64a63925161185d
./data/86:
86b25cef771f189ca565bc19c99ed6c7c9e3691ba3758079179a600c58e06f88
./data/ad:
ada0c8097a969bf9d126b39d84da8bfbf5dc428a354c31151c3e1cc86e742fce
./data/ae:
ae6b17714a70a19870c998ce0e9a28734f2ea0f471aec6832791dc8ce0ffaa69
./data/cf:
cfd2ce39624aa9a12d3656deecbf5ecc02144fc277b84e0b1e30669527e80d62
./index:
474bc50b2c62b12fc3ac8d7127c9583a6468f6e652bab263c8b78e14896f553f
a1243cf997162370e57ad2589f605271922748b20f8800b8090cfb1752b52285
a75ea15927343280ab9a4dfdbf6615f6c1489dc89c132f334f8f866159e0ac1b
e83ccb66f95576b834669f31fb7f8e69059a4e343eacba679c6941fba66df4a3
./keys:
f588b797f23be312426aecfe2d13473f4e0d86ee60cde3c5d6a30cf6b6259e44
./locks:
./snapshots:
104598c8811d078906e5f1d13044e83d826df034cfb3a1c86a31ac7b317d7876
61778ecfa0b209c6eb95a9c07a5058cf80bddb28e427a42e7f6495e377b06d61
74c189425790faf2497ddface2a041ba7fbb883223fa8c93235b8d1b3d7f5d2c
7d9cf8d381ff6651d5761f9fd8d15566d142e70d827380500b47708eaa4840c3
Commands like restic forget
or restic prune
can’t be used in that state and therefore we are most likely unable to recover from that error without deleting that whole repository.
If you need any more input, please just ask. If really needed we could also share one of those (corrupt) repositories including the PASSWORD file and also the source data of the volume. But that should be the last resort.
Thank you very much in advance for your help.
Bernhard