My use case: I use restic to backup all my work every two hours. I have constant workflow, office files changing every now and then so I want to keep all these versions. Files I work with are mostly MSOffice/LibreOffice files. As you may know, these files create .~lock. files when they are in use. I exclude those lock files from backup. As far as my testing skills allowed me, I noticed that restic still works when a file is in use, backing up the file-in-use until that point or the latest saved version of the file. For example, if I open LibreOffice Calc, edit a cell and then press enter, then restic save the file correctly until that point (after the file refresh itself) but if you’re editing a cell when the backup occur, then restic backup the file but without those changes. This is totally understandable and coherent.
My real question is, should I temporarily exclude those in-use-files from a backup preventing, maybe backing a corrupted file or excluding them is a little excessive/unnecessary work? Thanks in advance for your help.
If you are on Linux, my suggestion would be to make use of LVM snapshots to take your backup, and not exclude lock files.
An LVM snapshot is an atomic point-in-time copy of an entire filesystem, which effectively simulates a power cut to the machine. If you wind up having to restore the file from the backup, also restore the lockfile – then LibreOffice’s own crash recovery process will kick in, and restore any partially-written changes.
If you do not use an LVM snapshot and the backup runs while LibreOffice is writing changes to the primary document file, you might capture different regions of the file before and after the changes are saved, resulting in a file that’s half one set of changes and half the other set. This file may be completely unreadable. The lock file, I believe, includes a journal of changes that are pending a disk write.
The tl;dr is that using LVM snapshots will work with any software that is able to recover from an unexpected power loss.
Noted. I will delete this exclude. Thanks for pointing this out.
I’m using Linux Mint 19. I don’t really care much about backing up the system because at least in my case, it is pretty much standard and it is pretty easy to just start over (and I do prefer to do a clean install) in another machine as long as I have my files and just one VM that I already am backing up with restic (VM is not used that much, so if it is running, the backup process will not execute). As for the system, I have timeshift running daily (it was already installed with Mint) just in case I need to go back to a previous state.
What benefits a LVM snapshot can give me VS my current setup?
Then if I’m just using restic (not LVM) in this scenario, for what you’re saying then truncated files could happen (I used “corrupted” word before and I think that was incorrect) but it is possible to recover a file if I restore the file and its lock file, right?
This whole existential problem started because I restored a truncated file and I was not able to recover it (lock file was present and didn’t help). Well, I do recover it using an older version but latest changes were lost. At least I remembered the changes and it was not a lot of work. That’s what made me think, I prefer to temporarily exclude a file than backing up a damaged one. Does that seems reasonable or I’m just worrying myself for something that is not really worth to worry?
Thanks again for the LVM suggestion. I have vaguely read about it but didn’t really paid much attention to it. I will look up more info about it.
If there is any application activity during a backup, not using an LVM snapshot risks creating a backup where the files being modified are corrupt.
So if you do your backups at 3AM and you don’t use your computer at 3AM, then your document backups are almost certainly going to be fine.
It is sometimes possible, assuming that the backup was not taken while LibreOffice was in the middle of saving a change to either the lock file or the document file.
Consider the case where a single change requires modifying two separate regions of the document file. Restic could come in and take a backup after one write operation but before the other, and wind up with a file that’s half one version and half another version. Such a file may not load successfully.
This interaction is also possible (and even more likely) across files.
There’s really only a few ways that “save” can be implemented:
Truncate the target file and write out the new contents. This risks data loss if the power is cut during the write operation(s). Backups taken during the write operations will store a bad version of this file.
Write out a new file and, when fully saved, move it over top of the existing file. A backup sees either the old or the new, but not both (moving a file, even on top of another file, is an atomic operation on Linux when the source and target are on the same filesystem). This is the easiest approach to get correct, but requires that the filesystem have enough free space to store the “new” file. It can also be wasteful if the file is very large but only a small change is being made.
Compute what changes need to be made and modify the file in-place. This, like option 1, risks data loss because it’s possible for the program to apply some of the changes and then a power cut prevents the other set of changes from being made. In the event that the changes are interdependent, the file becomes corrupt. In the event that the changes are not interdependent, you wind up with a file that is valid but contains some data from the original state and some data from the new state.
Option 3, but additionally use some type of journal (write-ahead OR rollback).
A WAL (write-ahead log) is implemented as: (1) write out what changes we will make to another file, (2) flush this file to disk, (3) make the changes to the main file, (4) mark the WAL entry as completed. If a power cut happens, the program first consults the WAL to see if there is a valid entry that was not completed; if so, it uses the WAL to complete the change that was in progress.
A rollback log does the same thing except it copies the sections of the original file that are about to be changed into the journal. Then, if a power cut happens, the program can restore the file to its state before the change was attempted.
It really depends which approach LibreOffice uses, and I don’t know enough to say for sure. However, I do know there is some mechanism of crash prevention.
For other programs (think database servers) where one conceptual data set may span files, this problem is significantly exacerbated, as multiple files may need to be changed as part of a single update operation. Restic has no hope to accurately capture a single moment in time against a multi-file data set that is constantly changing.
For save approach 1, there is not much you can do because the application itself can’t deal with a power cut. The only sensible approach is to terminate the application before backing up.
For save approach 2, restic is fully capable of handling on its own without an LVM snapshot. Which file restic saves is decided at the moment restic opens the file. From that point on, it is either dealing with the old file or the new file, and will never see the other.
For save approach 3, similar to 1, the only safe approach is to close the application before backing up.
For save approach 4, you either have to make sure the application isn’t running OR use an LVM snapshot. A snapshot works because it simulates a power cut, and gives restic access to the point-in-time state of the entire filesystem. If the application was caught mid-write, the journal will give it enough information to complete or roll back the change when restoring from the backup.
Thank you very much for your explanation. I really appreciate it. It made me look up more info about LVM snapshots but I think (IMHO) LVM are useful for more complicated setups than mine, since I don’t really have a lot in my system that really needs this kind of solution, just a bunch of documents in Calc/Excel, Writer/Word, PDF’s, etc., I think I will create some kind of temp exclude for these files; when I say “temp” is to use find to see if there are lock files and exclude the open document. For example:
restic backup $backupdir --exclude-file="myexcludes.txt" --exclude-file=<(find "$backupdir" -iname .~lock.* 2> /dev/null | sed -e 's/.~lock.//g' | sed -e 's/#//g') --exclude=".~lock.*"
LVM is a great solution but for my specific use case I prefer a simpler solution. After all, I’m just backing up a bunch of documents to a “cloud” service.