Hi together,
I am working to get velero my k3s cluster working.
I am using OpenEBS Local PV / HostPath. The FSB (restic) does it’s thing partially.
I get alot of errors as follows:
The restic error code is:
restic backup --repo=s3:http://minio:9000/velero/restic/nextcloud --password-file=/tmp/credentials/velero/velero-repo-credentials-repository-password --cache-dir=/scratch/.cache/restic . --tag=backup=backup1 --tag=backup-uid=4ed0aac7-c8bb-4c70-a45e-1863ff79d653 --tag=ns=nextcloud --tag=pod=nextcloud-6c96775469-bp2bh --tag=pod-uid=6ba5f289-ccac-487f-95e4-4efaa76f68c7 --tag=pvc-uid=908abdc1-4280-4cd7-9500-1a09dbcc6efb --tag=volume=nextcloud-data --host=velero --json with error: exit status 3 stderr: {"message_type":"error","error":{"Op":"read","Path":"data/username/files/SofortUpload/Camera/IMG_20221120_095648.jpg","Err":5},"during":"archival","item":"data/username/files/SofortUpload/Camera/IMG_20221120_095648.jpg"}
It is a nextcloud deployment which I want to backup. But those files aren’t backed up.
Velero installed via Helm, Helm Chart Version 3.1.4
Velero Version: 1.10.2
Restic Version: 14.0
Any suggestions?
The more readable version of this error is:
read data/username/files/SofortUpload/Camera/IMG_20221120_095648.jpg: input/output error. That usually indicates a problem with the underlying storage from which restic is trying to read.
@MichaelEischer thank you for your answer. I did found a “solution” for this error.
Restic is called by velero to create PersitentVolume backups of pods. I use a pre-backup hook to freeze the FS of this PV.
The tool I use is fsfreeze.
Because the PersitentVolume is just a hostPath fsfreeze freezes not only the the PV but also the underlying filesystem. Which then has some side effects, like Pods/Containers are dying while there are not able to write to the same underlying Filesystem.
It is maybe not a actuall restic problem but more of a problem in the implementation of Velero. But I do not understand why the error happens. The actuall files are backupped at night and those files are nextcloud files, which are not used/modified or anything else.
Thank you.
Judging from the fsfreeze manpage, it is not possible to access files on a frozen filesystem.
Actually if I am using fsfreeze restic doesn’t throw this error, only without fsfreeze the error occurred.
If a filesystem is in freeze state, you can not write new files nor modify files. But reading files does work.