I use restic to backup my server every hour or two. This means that there are two category of “errors”:
transient: ones that touch single files that were probably used at the moment of the backup:
{
"message_type": "error",
"error": {
"message": "incomplete metadata for /var/lib/docker/volumes/technitium_config/_data/apps/Query Logs (Sqlite)/querylogs.db-journal: xattr.list /var/lib/docker/volumes/technitium_config/_data/apps/Query Logs (Sqlite)/querylogs.db-journal: no such file or directory"
},
"during": "archival",
"item": "/var/lib/docker/volumes/technitium_config/_data/apps/Query Logs (Sqlite)/querylogs.db-journal"
}
These will probably be cleared at a future backup pass
systemic: errors that put in danger the whole backup (I did not get these so far, but I imagine that it could be for instance some kind of inability to write to the repo or something similar)
My backup manager rises alerts for all errors and I would like to propose to add an extra field to the error message such as nature, that could initially have the values of transient and systemic (or whatever) – this would help a lot to discard the ones that will be fixed on a future pass happening “soon”.
How would restic determine what’s transient and what’s systemic?
The inability to stat a file might be because it was there at the start of the backup but then was removed as it’s a temp file, or it could be a permissions issue on a key file you really, really want to backup?
I don’t mean to suggest your idea isn’t valid - only that I don’t think it’s the role of a backup solution to decide what’s an “OK” error vs a “You need to fix this NOW” - so I’m curious how/what you’d use to classify?
My classification was based off the idea that some errors mean “the whole backup is borked” vs some others that say “a file could not be baked up”. The first case this requires an immediate reaction, while the other, in my case, is a weaker signal (possibly tracked and raised if it is repeated → deal with it or exclude it)
I guess that my question is more “are there errors such as the ones in the question that signal an issue with the whole backup? (as opposed related to a single file)”
Perhaps reviewing the exit code is a solution to the problem you’re apparently trying to solve.
Exit code 3 would mean “some files could not be backed up”. Other exit codes mean different things, with different levels of severity.
If you’re certain this is a “weaker signal” and doesn’t require your attention, you could perhaps have your backup manager ignore errors from restic backup commands that exit with code 3. I’d do this with caution though.
“A file could not be backed up” may be a minor inconvenience that will resolve itself, or it may be severe issue requiring immediate intervention. Restic has no way to know whether some hypothetical future backup will resolve the problem satisfactorily. This means the severity of this error is entirely dependant upon the individual scenario and dataset.
For example: I might not care that restic is never able to backup some temporary file that continually gets created/destroyed. However I might care very much that it is unable to ever backup a family photo because of a permissions error.
Old story… restic is not a full OS backup software. It is a very good software to backup some files - that’s it.
There are two option here.
Either use some other software which is the right tool for the job. Something like Veeam Backup maybe?
Or. Build your own full system backup solution which restic will be part of. I am afraid that what you are doing now is just an attempt to backup full system without any guarantee that you will ever be able to restore your system from it.
It is not by any means correct that your backup fails to read (and backup) some files. As what? You are OK that after restore your OS will have some random files missing? What’s the point?
Most likely reason is that you are trying to run it against live system.
As a minimum what you need is a stable source - snapshot (BTRFS/ZFS/LVM). Maybe you have to stop some services before taking snapshot to ensure consistency? No idea. Depends on what software you are using.
Or just focus on backing up what is critical - your data, config files etc. Reinstalling clean OS from scratch is few minutes job.
No, I am not trying to backup the OS - I run everything in containers so the host OS does not really matter
There are files that are being used during the backup so it is normal they fail at that moment. In my case this is a transient situation and the next backup will manage it.
Yes, that would be better, but in my case this does not matter that much.