Encountered the following error after running the ‘restic check’ command for restic version 0.13.1.
repository a157a0f0 opened successfully, password is correct
created new cache in /tmp/restic-check-cache-2364813881
create exclusive lock for repository
load indexes
check all packs
pack b3db8925: does not exist
check snapshots, trees and blobs
Load(<data/b3db892515>, 0, 0) returned error, retrying after 552.330144ms: open /safe/data/data/b3/b3db8925156744250db51fdb13cb1955bad94accdaf78b20aacc23072ed61edb: no such file or directory
...
Load(<data/b3db892515>, 2263, 5170) returned error, retrying after 27.337308488s: open /safe/data/data/b3/b3db8925156744250db51fdb13cb1955bad94accdaf78b20aacc23072ed61edb: no such file or directory
[4:18] 100.00% 22 / 22 snapshots...
Fatal: repository contains errors
I was wondering what could be the possible reasons for this error, and what would be the recommended solution. It’s worth noting that after running restic rebuild-index the error disappeared. Here is the output after running restic rebuild-index:
enter password for repository:
repository a157a0f0 opened successfully, password is correct
loading indexes...
getting pack files to read...
removing not found pack file b3db8925156744250db51fdb13cb1955bad94accdaf78b20aacc23072ed61edb
rebuilding index
[0:41] 100.00% 2211739 / 2211739 packs processed...
deleting obsolete index files
[0:00] 100.00% 170 / 170 files deleted...
done
Generally speaking, there could be different reasons for that, but the most obvious is that the file was lost in your filesystem.
However, you haven’t provided relevant information, so we can only guess. Anyway, I highly recommend that you upgrade to the latest version of restic, which is 0.16.0 at the time of writing. And if you’re using rest-server, you should upgrade that to 0.12.1.
I guess the repository is stored on a local disk? Which filesystem is used for that disk?
Did a restic run fail prior to the check error? Was that data file the only one missing? Did you notice anything unusual (e.g. system crash, USB drive not detached properly?). How often do you run check?
Did you remove anything other than the repeated errors that start with Load(<data/b3db892515> from the check output?
Does restic check --read-data complain about anything? Please also run fsck for the filesystem on which the repository is stored.
Restic only writes an index after asking the OS to actually write the corresponding data files to disk. Unless there’s a bug hidden in that part of restic (I’m not aware of anything in that regard), then the only other option is that something outside restic’s control did not behave as expected.
Thank you for your reply. We use NFS and run the backup through jobs in Kubernetes. After completing each backup, we run"restic prune" and “restic check”. The backup completes successfully but the prune fails. It’s possible that a restic run might have failed before the check error. We’re currently monitoring for issues like system crash. So, does this mean that a system crash can lead to such issues? Could this issue be related to NFS? Is it possible that “restic check --read-data” and “restic check” produce different outputs? .
Maybe, that depends on whether the NFS shares correctly process fsync or not. If fsyncs are processed and ensure that the files written by restic are properly stored on disk, then there won’t be a problem with system crashes. If fsyncs are ignored by the NFS share, then data may be lost if the storage server and/or the client crashes.
--read-data enables additional checks. The pack does not exist part won’t differ between both variants.