"The specified key does not exist" after S3 storage backend crash

1

We are using restic with Netapp S3 storagegrid as storage backend.
This storagegrid has had a hardware failure and we changed the controller.
Netapp told us that there should be no data lost, but restic now reports a
lot of errors:

root@fex:# restic check
using temporary cache in /local/tmp/restic-check-cache-598160385
repository 13e662d4 opened (version 2, compression level auto)
created new cache in /local/tmp/restic-check-cache-598160385
create exclusive lock for repository
load indexes
Load(<index/7c44af6b46>, 0, 0) returned error, retrying after 547.80443ms: The specified key does not exist.
Load(<index/5aa50131a8>, 0, 0) returned error, retrying after 526.079973ms: The specified key does not exist.
Load(<index/8159d3850b>, 0, 0) returned error, retrying after 350.72184ms: The specified key does not exist.
Load(<index/8159d3850b>, 0, 0) returned error, retrying after 391.495728ms: The specified key does not exist.
Load(<index/5aa50131a8>, 0, 0) returned error, retrying after 716.405635ms: The specified key does not exist.
Load(<index/7c44af6b46>, 0, 0) returned error, retrying after 582.465802ms: The specified key does not exist.
Load(<index/8159d3850b>, 0, 0) returned error, retrying after 688.692619ms: The specified key does not exist.
Load(<index/7c44af6b46>, 0, 0) returned error, retrying after 1.42724539s: The specified key does not exist.
Load(<index/5aa50131a8>, 0, 0) returned error, retrying after 867.706878ms: The specified key does not exist.
Load(<index/8159d3850b>, 0, 0) returned error, retrying after 1.996518187s: The specified key does not exist.
Load(<index/5aa50131a8>, 0, 0) returned error, retrying after 2.364653266s: The specified key does not exist.
Load(<index/7c44af6b46>, 0, 0) returned error, retrying after 1.134568167s: The specified key does not exist.
Load(<index/8159d3850b>, 0, 0) returned error, retrying after 3.13138797s: The specified key does not exist.
Load(<index/7c44af6b46>, 0, 0) returned error, retrying after 3.538372387s: The specified key does not exist.
Load(<index/5aa50131a8>, 0, 0) returned error, retrying after 2.384585694s: The specified key does not exist.
Load(<index/8159d3850b>, 0, 0) returned error, retrying after 5.168265712s: The specified key does not exist.
Load(<index/5aa50131a8>, 0, 0) returned error, retrying after 4.216853121s: The specified key does not exist.
Load(<index/7c44af6b46>, 0, 0) returned error, retrying after 4.19172784s: The specified key does not exist.
Load(<index/5aa50131a8>, 0, 0) returned error, retrying after 6.097058872s: The specified key does not exist.
Load(<index/7c44af6b46>, 0, 0) returned error, retrying after 7.337440995s: The specified key does not exist.
Load(<index/8159d3850b>, 0, 0) returned error, retrying after 5.384673948s: The specified key does not exist.
Load(<index/5aa50131a8>, 0, 0) returned error, retrying after 10.768539226s: The specified key does not exist.
Load(<index/8159d3850b>, 0, 0) returned error, retrying after 6.527470665s: The specified key does not exist.
Load(<index/7c44af6b46>, 0, 0) returned error, retrying after 9.543433219s: The specified key does not exist.
Load(<index/8159d3850b>, 0, 0) returned error, retrying after 14.931060051s: The specified key does not exist.
Load(<index/5aa50131a8>, 0, 0) returned error, retrying after 13.355364696s: The specified key does not exist.
Load(<index/7c44af6b46>, 0, 0) returned error, retrying after 10.446366265s: The specified key does not exist.
Load(<index/8159d3850b>, 0, 0) returned error, retrying after 24.568262527s: The specified key does not exist.
Load(<index/7c44af6b46>, 0, 0) returned error, retrying after 25.368385098s: The specified key does not exist.
Load(<index/5aa50131a8>, 0, 0) returned error, retrying after 19.896642553s: The specified key does not exist.
[1:04] 100.00%  6 / 6 index files loaded
error: error loading index 5aa50131a82f0dfc29ed1e0b7710208ab3e676b8f543e4b488ecede2e3c19e6c: The specified key does not exist.
error: error loading index 8159d3850bfc8bf2df64b7ff850ce24e46c3a1248f776131e45732c13a53d472: The specified key does not exist.
error: error loading index 7c44af6b460e536e02febf976d7589d44f0e8f90da5381f35ab6c04b77477da6: The specified key does not exist.
Fatal: LoadIndex returned errors

root@fex:~/doc# restic check
using temporary cache in /local/tmp/restic-check-cache-171821539
repository 13e662d4 opened (version 2, compression level auto)
created new cache in /local/tmp/restic-check-cache-171821539
create exclusive lock for repository
load indexes
[0:01] 100.00%  7 / 7 index files loaded
check all packs
pack 8345f3be71925331a476da043b267a9bc2293b59111424951b1c976e614f39f7: does not exist
pack ff0564345fd890b148a6079a87173dd3d7de9e991452750edde61332c2c6eeca: does not exist
pack 9a4cf543040820002577fcbf282cdd9ca5e6c14f34bfc582127efdb80814e6dc: does not exist
pack 79b2dc88c3af2f676a548208c6cd8bc074d76eac6d0e419acbe4a143ddf1bf8e: does not exist
pack 8352badad5b919b71fa00f28477462cbacc7d3717b5f349521c85631da184a91: does not exist
pack 7aa0343bd51f96549de201d00cbc49bc8eb00d6969e94fa995938541d5da0f01: does not exist
pack ff5e41e23885c6b5a1669d54a29173e2aa18a5dfd7187b8cc92c460bc7e65c89: does not exist
pack 799e8cceacf01ce8c6ffab2b7601a3b41f7196d0983b35ae1013f51f25225af2: does not exist
pack 7fa81abe7bb5b5b6713eb9a7b8dfe5bb1f3d0ca639cb69aec0bc7409d46067e9: does not exist
pack 9bc70a3d03f52675e3943e297bbb42673a1d948f6653606de4142f6cec5fa1cb: does not exist
pack 7b646ceb4b283b4c029be36f8516ae838d7ae774c315ae6dd6b7ec18073dc4f7: does not exist
pack 9d0a7c855fd53d6d08b46e41ca30321c938513fa00fbcf816b8017e3ad9a3424: does not exist
pack 9b66aac6a61a75922ef9f724ce8ea9b9be4de043e3b679ac80f6fc9af7364239: does not exist
pack 9a1d2c26af962774d10718ad7cb1e9b493fc8299e776734cd41dc53c932d9766: does not exist
pack 794ba9259be888beca5540ad84c1b1645a39021fe81c500c32d92adafbac0e97: does not exist
pack a088af8e17053bbbf83c17bdd6680e4e854045cba32c48578f094fb7960f2558: does not exist
pack 806fee5851febcba7e39a4e9abfa7c320a0b9e28855816df65ab9946cf638484: does not exist
pack 99ad07ef89a62aee217546ed45f22fbe9f152c5b1771fa25623658756e96ae3e: does not exist
pack 7a935751a462b098b2954930789d228f557ccbe960931a8ed2b17cf0d9c01915: does not exist
pack 9f0650bac3eaa14b92e1a6f29f7d5de85732d4bb8066fe12781fb9f00fa2e6a6: does not exist
pack 777c9b10c04221d4f5584ec8cf9e8abc222cfad4836ec68782e93f194ecc686b: does not exist
pack 7bbdf45c784c11390b9a3385473e49a998a89949fe19bbdeedd0201fa2725007: does not exist
pack 8072c30d0756d558f5f71c64aa6e774f8da0b16068557294ce7fa3b68b2ae62e: does not exist
pack 9976af4553d8a1fcd15b4370c399cb678dfd8125e45a1516d6470615f3441904: does not exist
pack 808bb8dbd60eab03b1a11373f3645debd09b1ff434a85a68ce5b1690b92d7443: does not exist
pack a027333f7c595bd578a1d9a9c1215afb2246e15d0d5daf96b39ce272586d507d: does not exist
pack 82e107c7bc28c5e90ec0d9a556d982d2ff33ce150b3090d52aba16da7000b45e: does not exist
pack 7d8f5aa321b4e597f38ac2d3602ca9d1cfae130f82a33d5cc038d06e590f51b1: does not exist
pack 8235a7afc9e9cb1c6d98da5303ad740cf08f8268d03c839299888e3f7db05ee4: does not exist
pack 9d8e6395acbb325462dc44db4891ccec7891b952bf37e89b7f3bb738c283d3b3: not referenced in any index
pack fe0685f9f40352e5c651be8032dfe2d1b0ceb6d6fe4569f6ba48febd0d9dc1f7: not referenced in any index
pack 769c936e7986fc58de6e85f3c8502657cfe112667680f525079a5cc65a046bd3: not referenced in any index
pack 781d47a3bc1ed5ba6941d23a03186547a05dc1f3c319c6265301ccfb454af534: not referenced in any index
pack 841ff7223df7947cf08c2d6c601d1f4e7c0e6e56250c55817ea20efe9c3691ec: not referenced in any index
pack fe555eea14fe13fe2cf9ba5151cecb13fe8fbb1df77033b6d28a651c6d254879: not referenced in any index
pack 9b6c357f3019b37a80aaa9c708672b7afeddc1b5df3c320d4b27f6be9363d2a4: not referenced in any index
pack 8378ff4223ac579225b7a308f84dd4ca503f74a90b613c362d7082fdbe4e1df6: not referenced in any index
pack 783895402ed554dd8b0859d1a3c8bf0982b7108e7363dd85095251288b2f7f25: not referenced in any index
pack fd5a488f24ce861bf191a1f9c7d6d92333fd3d1b5b8a92436d71224c523c0484: not referenced in any index
pack 7f8fff6a0d1ada4ef17c3afd5e364fb1f37545dfd44787ed607b6d4f36e038ae: not referenced in any index
pack 7ee2b07ee7421e86ab143dc8d389f81b0ad8a595dadcd74ad4c8555fc53b4c9e: not referenced in any index
26 additional files were found in the repo, which likely contain duplicate data.
This is non-critical, you can run `restic prune` to correct this.
check snapshots, trees and blobs
[0:07] 100.00%  27 / 27 snapshots
Fatal: repository contains errors

root@fex:~/doc# restic prune
repository 13e662d4 opened (version 2, compression level auto)
loading indexes...
Load(<index/d3b6fba40b>, 0, 0) returned error, retrying after 643.849989ms: The specified key does not exist.
Load(<index/d3b6fba40b>, 0, 0) returned error, retrying after 389.779958ms: The specified key does not exist.
Load(<index/d3b6fba40b>, 0, 0) returned error, retrying after 1.523605531s: The specified key does not exist.
Load(<index/d3b6fba40b>, 0, 0) returned error, retrying after 1.892593422s: The specified key does not exist.
Load(<index/d3b6fba40b>, 0, 0) returned error, retrying after 1.35946936s: The specified key does not exist.
Load(<index/d3b6fba40b>, 0, 0) returned error, retrying after 2.43941728s: The specified key does not exist.
Load(<index/d3b6fba40b>, 0, 0) returned error, retrying after 3.679355455s: The specified key does not exist.
Load(<index/d3b6fba40b>, 0, 0) returned error, retrying after 5.566907485s: The specified key does not exist.
Load(<index/d3b6fba40b>, 0, 0) returned error, retrying after 13.290047392s: The specified key does not exist.
Load(<index/d3b6fba40b>, 0, 0) returned error, retrying after 26.251447542s: The specified key does not exist.
[0:57] 87.50%  7 / 8 index files loaded
The specified key does not exist.


root@fex:# restic version
restic 0.16.4 compiled with go1.21.6 on linux/amd64

What can I do to correct these errors?

2

My storage admin told me, that a synchronization job is running on the storagegrid cluster which makes it pretty slow.
Maybe these errors are because of timing problems?!

3

And now two “restic check” together with only a few seconds delay:

root@fex:~/doc# restic check
using temporary cache in /local/tmp/restic-check-cache-3506090368
repository 13e662d4 opened (version 2, compression level auto)
created new cache in /local/tmp/restic-check-cache-3506090368
create exclusive lock for repository
load indexes
[0:03] 100.00%  7 / 7 index files loaded
check all packs
check snapshots, trees and blobs
[0:07] 100.00%  27 / 27 snapshots
no errors were found


root@fex:~/doc# restic check
using temporary cache in /local/tmp/restic-check-cache-3679478365
repository 13e662d4 opened (version 2, compression level auto)
created new cache in /local/tmp/restic-check-cache-3679478365
create exclusive lock for repository
load indexes
[0:01] 100.00%  7 / 7 index files loaded
check all packs
pack db2bdd53e67934b379baee7faa4eec59a393209eccd21bf7ca6d408d5ef42a42: does not exist
pack de910351d9d12845c531a5e385e802331ca538b83483428ccbb45ee2faa3e202: does not exist
pack b0a1ee46128fe90f77a483eeb5d5a8d7e72e4d68d5beb86520b022c31ba2911c: does not exist
pack afa0c8c24feb2f7088e96c65d25aa49ddcc92843b379b9af11673e5445c81afb: does not exist
pack b1f7f2b59c61aef5c400d099ce985e9505ecbdfbdb3fb22852d9f40a80ed2653: does not exist
pack b587d4eb14d25f2e2cc9831f974ec35a58941269cf6a539040ac079a1cb13cbe: does not exist
pack dae0516b9f526a89be57961a74900d90e9a1faca86dae20ba9efc5403e06250b: does not exist
pack dc152219dc319b3004f363815537187207840e2788785d632987797ae147e9e1: does not exist
pack 1513c89cc0ec408c0c3dfd5131c4af27079242043f9bcc8caa53680a1fd073fe: does not exist
pack ddc150b1ac19a4f7848415e17beaea2ff679e218c379c8eac0be419378616c61: does not exist
pack b339ad70c2aa03c72fc5c8fc4640854d483c09bf35dd4f127c1f0fc4c1b31fca: does not exist
pack da7a60fea5a8e0978c7aaeabcebc15c6a368b8efdcacc16ab7e95e0874584b40: does not exist
pack decb8377fa8c899ef545680dd78da62b98a21b2aa075679cd89e468f4a1c82d0: does not exist
pack db46e10ab586edc9e5d882c6bccdac777d0f18aced378130cc290e2a13a2ce56: does not exist
pack 112f5a25fc8776a5677c73ef0d0532e297b887a94266142dc6d7904d3b9b021e: not referenced in any index
pack b0d2abb956b2ebdd0cf18b0dffaf312c2fb7663f89a80d59c3ab994ff9dda23e: not referenced in any index
pack b3398217df2b4b6115256737ebf0a52eb5f6cf7a7a3780b8008f7da0ef2609bd: not referenced in any index
pack ddb68cbb5c2c7f798833a987ad278273405d0b8e1549048cffef8e3d60493947: not referenced in any index
pack b2691c91729e005e89268a2a23796df7faaccb2196e14fba6decfa50511449e8: not referenced in any index
pack b147a478b838a6d2cade7a3a914b9bb154d36958c9d606575fabcefc788dd3e4: not referenced in any index
pack da2f8303ebef612fcc8b5d477141583f338022b5ed864192d243d5e4020497c2: not referenced in any index
pack b19227ca4dab7a8c06cc2afff652077a83125051052bc7a5ca455ac1d0b48927: not referenced in any index
pack afebe915dd3667c602a4ac53a6ad1d466a0fdfb540dc8d01633ca96c144b52dc: not referenced in any index
pack 14d9edc67a24e92a862deaa5f6d92e74d37ee522d41c007afb2aea52f1a826fe: not referenced in any index
pack b27db7f57c6287501030ed91368fb263b55e8e7d6b045e7ecdd506102aff4290: not referenced in any index
pack b38f2aa2e396135ed9930b3794365a060e403a89c243f11a6c3c5988b655b2a0: not referenced in any index
pack dacea057fc45cdfd1696163128193116f4bdacd24868bb70a5ce16269e71b38c: not referenced in any index
pack dbcb1351ce6f17cc7bb4e4e5ebbd713d43389c777125000baec2c7a69d40d48e: not referenced in any index
pack dbb19e96fc194dbe72930f8673116f66668e22268d11383a0a6b830b37ccf5f1: not referenced in any index
pack b335a34371ec75a3933e4c82911005d5546a0a735221dd661a465439db96bb5e: not referenced in any index
pack b1aab2e6e03a712e2b7bb797aceba4677b1aae0f55f78b5e074c116e2614ea0d: not referenced in any index
pack 0fbff0f65d3e9e4af0ac0809b669208b9ca54aba557f07872e309666ddb97aa4: not referenced in any index
pack b0cd5eb8ffaf0d0a0c2dacb7d4f41de2eda5852ec4818cd1701b501d1b73771f: not referenced in any index
19 additional files were found in the repo, which likely contain duplicate data.
This is non-critical, you can run `restic prune` to correct this.
check snapshots, trees and blobs
[0:06] 100.00%  27 / 27 snapshots
Fatal: repository contains errors
4

That usually means that the file listing returned by the storage and the actually existing files are out of sync. I hasn’t anything to do with timing. The synchronization job is probably working on sorting that out.

The storage server still seems to be out of sync. Apparently, some server is not yet up to date (not sure how the storagegrid architecture looks like).

5

My storage admin told me, that the storagegrid has two mirrored servers, one was
replaced (after a hardware failure) and the resyncing will last about 4 days.

Maybe this is the cause of the error messages and I just have to wait until
the resyncing has finished?

This means I do not have access to the backup until then.
Bad situation…

6

One day later everything looks ok, now!

root@fex:~# restic check
using temporary cache in /local/tmp/restic-check-cache-2656039474
repository 13e662d4 opened (version 2, compression level auto)
created new cache in /local/tmp/restic-check-cache-2656039474
create exclusive lock for repository
load indexes
[0:01] 100.00%  7 / 7 index files loaded
check all packs
check snapshots, trees and blobs
[0:06] 100.00%  27 / 27 snapshots
no errors were found

It looks it was a temporary failure of our Netapp storagegrid backend?

7

That’s consistent with the symptoms.

Yes, all errors reported by restic were backend problems.