I am testing restic and I am thinking of migrating my backup scripts that use rsync to restic. The rsync logs indicate the date the file was copied, but the restic seems not to.
Iâm not sure if I understand correctly. Do you want to know when file x was backupped the last time? Technically, this would not provide useful auditing information as a file is just included in a new snapshot if it has changed. And if it does get added and listed in your log file, you can add the date of the restic run before executing restic.
Yes.
I have 25TB of data with more than 3 million files. The transfer speed of the files is 15GB / hour, so the estimate to complete the transfer is approximately 70 days. Rsync informs the date on which the transfer takes place. Why canât restic be similar to rsync?
Humm⌠I do not understand why it is important to know exactly when (in the 70 day period) a certain file was copied.
Why not just let the first backup take itâs time. Then immediately start another backup. It will be much faster, since only changes will be uploaded.
When that second backup is done, you know that the backup is up to date.
You donât have to understand. It is a necessity of my backup transaction. A file, in case of recovery for any reason, may or may not be valid depending on how long it was for the backup. My question is whether or not restic can include this information in the logs just like rsync does.
Well⌠You came here asking questions, two of us wrote that we did not understand certain things in what you wrote. I find it a bit strange that you write that I donât have to understand. Understanding is the first step towards (maybe) being able to help.
Anyways⌠I still stand by my suggestion. Until the SECOND backup has been made, it seems like your need of knowing backup time of individual files cannot be satisfied. But subsequent backups are fast given that they only transmit the differences, so hopefully the window of uncertainty (the backup time) is small enough for you.
My English is awful. I use tractors to try to communicate. Sometimes it is difficult to make yourself understood. My question was about the possibility of restic indicating the dates on the logs just like rsync does. Regarding your suggestion, yes, I will have an estimated start and end time, but I would really like to be more precise about the date of sending the file. The time for the second backup will be shorter, but my data movement is very long and the data transfer will still take a few days. Thank you very much.
I donât think it can be done using only restic (even using the â-vvvâ flag), probably because it doesnât really make sense to have that information in standard use cases.
If I understand you correctly itâs about your having to know exactly which exact version of each file is included in a backup. Maybe you can work around the problem by using LVM to make a snapshot of your file system and then backup the files in that snapshot? That way they would all have been backed up at the exact same time.
No. It would be restic to have an option similar to rsync: rsync --log-file
man rsync:
âlog-file=FILE
This option causes rsync to log what it is doing to a file. This is similar to the logging that a daemon does, but can be requested for the client side and/or the server side of a non-daemon transfer. If specified as a client option, transfer logging will be enabled with a default format of â%i %n%Lâ. See the --log-file-format option if you wish to override this.
Here's a example command that requests the remote side to log what is happening:
rsync -av --remote-option=--log-file=/tmp/rlog src/ dest/
This is very useful if you need to debug why a connection is closing unexpectedly.
If the file transfer date information were in the log, just like rsync does, it would be easier.
And really restic has better attributes compared to rsync.
Unfortunately (in this case) this isnât the case, as others have established by now. The best you can do is some type of workaround to get the or close to the information you are looking for, e.g. what @dpfeilsticker suggested. I hope you can live with that, because it would be a shame if this was a dealbreaker for your use of restic!
Another type of workaround might be if the --json -vvv options outputs a line for each file when that file has completed backing up, and you can pipe this to something that picks out the filename and also adds a timestamp - if this works it should give you the timestamp for each file. Try it and let us know
rawtaz, thanks for the tip.
A stopgap I can use is to include the date when generating the log. #restic -vvv ⌠backup /data | while IFS= read -r line; do printf â%s %s\nâ â$(date â+%F %Tâ)â â$lineâ; done
It probably makes more sense to use a filesystem that supports atomic snapshots, like btrfs or ZFS, and back up from a snapshot. This is what I do on my systems. It assures a point-in-time backup similar to if you unplugged the power cable at the moment the snapshot is taken. This can be used to back up data for any application that can gracefully handle a power cut without requiring the application to be stopped.
About the solution @dpfeilsticker proposed: Note, that this doesnât show the information when the file was backupâed. It only shows the mtime of the file which restic does save of course. This info is available in the trees saved by restic (available e.g. with the cat command) and IIRC it is also restored by the restore command.
So, yes, it is a good solution to see what âstateâ of the file was actually backuped. But it is no solution to the original question which is not supported by restic. FYI, rustic supports the mentioned log format using the --log-file option.
Another thing to mention is backing up a filesystem with changing content is always risky in terms of consistency. A file could change just in between restic accessing the mtime and actually reading the content. If the file is modified while being read by restic, it depends a lot on the filesystem and operation system about whatâs gonna happening.
Also a logging info with time doesnât help much - youâll never know whatâs happening between actually reading the file and writing the log.
I think this is what the other posters tried to tell: If consistency is a issue (and for very long running backups, it definitively is), use a filesystem snapshot if available or redo the backup which will be much faster.
@Cal: See man ls - this isnât a restic specific command, it is about ls.
