Log format - rsync x restic

Hello. All right?

I am testing restic and I am thinking of migrating my backup scripts that use rsync to restic. The rsync logs indicate the date the file was copied, but the restic seems not to.

RSYNC:
2021/04/01 09:38:10 [64521] <f+++++++++ file01.txt
2021/04/01 09:38:11 [64521] <f+++++++++ filte02.txt

RESTIC:
new file01.txt, saved in 1.067s (460.559 KiB added)
new file02.txt, saved in 0.041s (754.452 KiB added)

For auditing it would be important to have this information. Is it possible to add synchronization date in the logs?

I’m not sure if I understand correctly. Do you want to know when file x was backupped the last time? Technically, this would not provide useful auditing information as a file is just included in a new snapshot if it has changed. And if it does get added and listed in your log file, you can add the date of the restic run before executing restic.

Yes.
I have 25TB of data with more than 3 million files. The transfer speed of the files is 15GB / hour, so the estimate to complete the transfer is approximately 70 days. Rsync informs the date on which the transfer takes place. Why can’t restic be similar to rsync?

Humm… I do not understand why it is important to know exactly when (in the 70 day period) a certain file was copied.
Why not just let the first backup take it’s time. Then immediately start another backup. It will be much faster, since only changes will be uploaded.
When that second backup is done, you know that the backup is up to date.

You don’t have to understand. It is a necessity of my backup transaction. A file, in case of recovery for any reason, may or may not be valid depending on how long it was for the backup. My question is whether or not restic can include this information in the logs just like rsync does.

Well… You came here asking questions, two of us wrote that we did not understand certain things in what you wrote. I find it a bit strange that you write that I don’t have to understand. Understanding is the first step towards (maybe) being able to help.

Anyways… I still stand by my suggestion. Until the SECOND backup has been made, it seems like your need of knowing backup time of individual files cannot be satisfied. But subsequent backups are fast given that they only transmit the differences, so hopefully the window of uncertainty (the backup time) is small enough for you.

Good luck!

1 Like

My English is awful. I use tractors to try to communicate. Sometimes it is difficult to make yourself understood. My question was about the possibility of restic indicating the dates on the logs just like rsync does. Regarding your suggestion, yes, I will have an estimated start and end time, but I would really like to be more precise about the date of sending the file. The time for the second backup will be shorter, but my data movement is very long and the data transfer will still take a few days. Thank you very much.

I don’t think it can be done using only restic (even using the “-vvv” flag), probably because it doesn’t really make sense to have that information in standard use cases.

If I understand you correctly it’s about your having to know exactly which exact version of each file is included in a backup. Maybe you can work around the problem by using LVM to make a snapshot of your file system and then backup the files in that snapshot? That way they would all have been backed up at the exact same time.

1 Like

No. It would be restic to have an option similar to rsync: rsync --log-file

man rsync:
–log-file=FILE
This option causes rsync to log what it is doing to a file. This is similar to the logging that a daemon does, but can be requested for the client side and/or the server side of a non-daemon transfer. If specified as a client option, transfer logging will be enabled with a default format of “%i %n%L”. See the --log-file-format option if you wish to override this.

          Here's a example command that requests the remote side to log what is happening:

              rsync -av --remote-option=--log-file=/tmp/rlog src/ dest/

          This is very useful if you need to debug why a connection is closing unexpectedly.

You can fuse mount your backup as you would do for restore:
https://restic.readthedocs.io/en/stable/050_restore.html

and with
ls -aRl --full-time /mnt/restic/
you will get a complete list of all files in all backups with exact timestamp

We switched from rsync to restic as rsync is terrible slow (especially if you make use of the --checksum option…)

If the file transfer date information were in the log, just like rsync does, it would be easier.
And really restic has better attributes compared to rsync.

Unfortunately (in this case) this isn’t the case, as others have established by now. The best you can do is some type of workaround to get the or close to the information you are looking for, e.g. what @dpfeilsticker suggested. I hope you can live with that, because it would be a shame if this was a dealbreaker for your use of restic!

Another type of workaround might be if the --json -vvv options outputs a line for each file when that file has completed backing up, and you can pipe this to something that picks out the filename and also adds a timestamp - if this works it should give you the timestamp for each file. Try it and let us know :slight_smile:

Here’s an example of a JSON line:

{"message_type":"verbose_status","action":"new","item":"/doc/man/restic-find.1","duration":0.000678317,"data_size":3878,"metadata_size":0,"total_files":0}

I’m not sure if this type of line is output when the file has been sent or if it’s output when the file has been scanned.

rawtaz, thanks for the tip.
A stopgap I can use is to include the date when generating the log.
#restic -vvv … backup /data | while IFS= read -r line; do printf ‘%s %s\n’ “$(date ‘+%F %T’)” “$line”; done

It probably makes more sense to use a filesystem that supports atomic snapshots, like btrfs or ZFS, and back up from a snapshot. This is what I do on my systems. It assures a point-in-time backup similar to if you unplugged the power cable at the moment the snapshot is taken. This can be used to back up data for any application that can gracefully handle a power cut without requiring the application to be stopped.

2 Likes

But my goal is to have the date of transfer of the file to the repository. It is just having a log with more information.