What would happen if a file changed while it was being backed up by Restic? In our case we are using Restic to move large (non-Restic) backup files to an offsite repository. These files can be 100-200GB in size.
What would happen if one of these files was altered/changed while a Restic backup was running? I suppose this question could also apply if we were backing up a live server and some changes were being written to a MySQL database, for example. How does Restic handle this?
Just trying to work out the best solution in terms of how/when to safely run Restic and ensure data consistency.
There’s nothing any backup client can do if your storage is not idle. Beside detecting that file changed or grew during backup, in which case you will be informed, here’s an example:
warning for /opt/observium/rrd/stats02/perf-pollermodule-status.rrd: file has changed
warning for /var/log/account/pacct: expected 122095744 bytes, saved 122098112 bytes
If you need 100% consistent backup, you need to provide idle storage during the backup run, typically with some kind of snapshotting (LVM or whatever).
Also databases should have their own backup procedures, and are best avoided when doing filesystem backup. You can for example do a mysqldump or xtrabackup dump, then backup this file/dump, and exclude /var/lib/mysql completely.
@zcalusic is right: When the files change while restic reads them, you’ll get an inconsistent state for the file, and the backup is worthless (depending on how well the application can recover).
For databases like MySQL, you could use mysqldump [...] | restic backup --stdin --stdin-filename dump.sql to archive the raw SQL stream.
@fd0, @zcalusic, or others:
In case a file / folder is inconsistent due to changes being made while restic backup is being run (e.g., via cron) – can the snapshot (and all the underlying folders) still be restored? Can the inconsistent files be restored (though into an inconsistent state of course)?
Yes, but for the files which were changed while restic read them, you’ll end up with an inconsistent state. The other files (and the folders) are not affected.
Yes, restic will restore them as best as is can. How good that is (and how usable the result would be) depends on the file and format and the changes made during the backup.
I have a follow up question on this because I’m evaluating using Restic to back up log files (which are frequently updated then).
So in my case the data written to the log files are immutable. And if there is some partially written log entries in the backup doesn’t matter. What is important though it that one the backup runs later these partially written log entries does not remain partially written (ie there is permanent inconsistency between the log file and the backup).
A scenario I can think of for example where something like this can occur is if the blob hash is not based on what is actually written to the backup or something like that (blob is read from file, hash calculated and then blob is read from file again when written to repo). Is this something that has been taken into consideration?
All somewhat recent versions of restic read files only once. That is the hash calculation always uses the exact same data as what is written to the repository.