I wrote a wrapper for restic, which makes an ssh to several machines, and for each of them does a backup, a check and an attempt to recover a few random files. Each machine does two such cycles, onto two different repos (so that I have truly independent backups in two dissociated places)
Every about 20 runs I get recovery errors on one specific repo (one that is on a remote machine with a USB drive plugged in, to which a specific server does one of its backups via the restic HTTP server)
srv→square: Restore verification failed: Restic exited with code 1:
ignoring error for /etc/docker/compose/domotique/esphome/.esphome/platformio/packages/tool-cmake/doc/cmake/html/_sources/policy/CMP0062.rst.txt: decrypting blob <data/e46ebfe2> from 4b0c69c8 failed: ciphertext verification failed
ignoring error for /etc/ansible/lib/python3.11/site-packages/ansible_collections/fortinet/fortimanager/plugins/modules/fmgr_system_admin_profile.py: decrypting blob <data/d62ae879> from 47f16708 failed: ciphertext verification failed
ignoring error for /etc/ansible/lib/python3.11/site-packages/ansible_collections/infoblox/nios_modules/plugins/modules/nios_range.py: decrypting blob <data/f6fe742d> from a5fca5ca failed: ciphertext verification failed
ignoring error for /etc/docker/compose/domotique/esphome/.esphome/platformio/packages/framework-arduinoespressif32-libs/esp32/include/espressif__esp_matter/connectedhomeip/connectedhomeip/zzz_generated/app-common/app-common/zap-generated/cluster-enums.h: decrypting blob <data/722244f9> from 1d1a3bed failed: ciphertext verification failed
ignoring error for /etc/ansible/lib/python3.11/site-packages/ansible_collections/fortinet/fortimanager/plugins/modules/fmgr_system_admin_profile.py: lchown /root/restic_verify_srv-square_20260715020013/etc/ansible/lib/python3.11/site-packages/ansible_collections/fortinet/fortimanager/plugins/modules/fmgr_system_admin_profile.py: no such file or directory
ignoring error for /etc/ansible/lib/python3.11/site-packages/ansible_collections/infoblox/nios_modules/plugins/modules/nios_range.py: lchown /root/restic_verify_srv-square_20260715020013/etc/ansible/lib/python3.11/site-packages/ansible_collections/infoblox/nios_modules/plugins/modules/nios_range.py: no such file or directory
ignoring error for /etc/docker/compose/domotique/esphome/.esphome/platformio/packages/framework-arduinoespressif32-libs/esp32/include/espressif__esp_matter/connectedhomeip/connectedhomeip/zzz_generated/app-common/app-common/zap-generated/cluster-enums.h: lchown /root/restic_verify_srv-square_20260715020013/etc/docker/compose/domotique/esphome/.esphome/platformio/packages/framework-arduinoespressif32-libs/esp32/include/espressif__esp_matter/connectedhomeip/connectedhomeip/zzz_generated/app-common/app-common/zap-generated/cluster-enums.h: no such file or directory
ignoring error for /etc/docker/compose/domotique/esphome/.esphome/platformio/packages/tool-cmake/doc/cmake/html/_sources/policy/CMP0062.rst.txt: lchown /root/restic_verify_srv-square_20260715020013/etc/docker/compose/domotique/esphome/.esphome/platformio/packages/tool-cmake/doc/cmake/html/_sources/policy/CMP0062.rst.txt: no such file or directory
Fatal: There were 8 errors
srv→square: Restore verification failed: Restic exited with code 1:
ignoring error for /etc/ansible/lib/python3.11/site-packages/ansible_collections/cisco/iosxr/plugins/action/interfaces.py: decrypting blob <data/a03a1c33> from 7a8df0fa failed: ciphertext verification failed
ignoring error for /etc/docker/compose/domotique/esphome/.esphome/build/atom-echo/.piolibdeps/atom-echo/libsodium/libsodium/src/libsodium/crypto_sign/ed25519/ref10/sign.c: decrypting blob <data/b032204b> from 059f5159 failed: ciphertext verification failed
ignoring error for /etc/docker/compose/domotique/esphome/.esphome/platformio/packages/tool-cmake/share/cmake-3.30/Modules/CheckOBJCXXSourceRuns.cmake: decrypting blob <data/de7b2635> from 45c73d51 failed: ciphertext verification failed
ignoring error for /var/lib/docker/volumes/technitium_config/_data/stats/2026053004.stat: decrypting blob <data/66597643> from ce3adbad failed: ciphertext verification failed
ignoring error for /etc/ansible/lib/python3.11/site-packages/ansible_collections/cisco/iosxr/plugins/action/interfaces.py: lchown /root/restic_verify_srv-square_20260715101540/etc/ansible/lib/python3.11/site-packages/ansible_collections/cisco/iosxr/plugins/action/interfaces.py: no such file or directory
ignoring error for /etc/docker/compose/domotique/esphome/.esphome/build/atom-echo/.piolibdeps/atom-echo/libsodium/libsodium/src/libsodium/crypto_sign/ed25519/ref10/sign.c: lchown /root/restic_verify_srv-square_20260715101540/etc/docker/compose/domotique/esphome/.esphome/build/atom-echo/.piolibdeps/atom-echo/libsodium/libsodium/src/libsodium/crypto_sign/ed25519/ref10/sign.c: no such file or directory
ignoring error for /etc/docker/compose/domotique/esphome/.esphome/platformio/packages/tool-cmake/share/cmake-3.30/Modules/CheckOBJCXXSourceRuns.cmake: lchown /root/restic_verify_srv-square_20260715101540/etc/docker/compose/domotique/esphome/.esphome/platformio/packages/tool-cmake/share/cmake-3.30/Modules/CheckOBJCXXSourceRuns.cmake: no such file or directory
ignoring error for /var/lib/docker/volumes/technitium_config/_data/stats/2026053004.stat: lchown /root/restic_verify_srv-square_20260715101540/var/lib/docker/volumes/technitium_config/_data/stats/2026053004.stat: no such file or directory
Fatal: There were 8 errors
There are two kinds of errors: ciphertext verification failed and no such file or directory
My first thought was “the repo is invalid for some reason”. So I deleted it, and let it recreate. The error came back.
I then thought “maybe the filesystem on which I back up is defective” – I removed the partition, recreated an ext4 filesystem, then the repo – same error again.
I am now at the stage of wondering whether the USB drive is not failing (it is fine according to SMART and ns fsck, but you never know).
Before I change the disk (which requires some preparation), I wanted to ask what could be (or is, or usually is) the reason for these two errors when trying to recover?
A check of the repo is OK
root@square ~# ./restic -r /data/backup/restic/srv check
using temporary cache in /tmp/restic-check-cache-2690532154
create exclusive lock for repository
repository 9edb19d9 opened (version 2, compression level auto)
created new cache in /tmp/restic-check-cache-2690532154
load indexes
[0:00] 100.00% 11 / 11 index files loaded
check all packs
check snapshots, trees and blobs
[0:03] 100.00% 32 / 32 snapshots
no errors were found
The srv machine mentioned in the errors also backs up the same data to itself (on a specific disk) and there are no issues.