I’m using Restic (thanks!) since a while. My content is about 250k files on aprox 2.5 Tb. Let’s say that 125k or so, are sidecard files (small text files) that changes (almost the timestamp), each time I run a backup. Rest of the files are video/photos.
Tooks more than 4 hours for creating a new snapshot. I think biggest part of this backup comes on the small files.
Anyway, surely there are some recommendations about how to speed up this backup.
Thanks a lot.
(PD: I’m working on OSX 10.15 with SSD and 24 Gb of RAM).
I think it would be helpful if you would provide some details about your operation:
What version of restic?
What are you backing up to (local drive? remote drive? cloud service?)
What is your command-line?
The entire output of the restic operation (especially the part that shows vita statistics: time, bytes processed, file stats new/changed/unmodified/added to the repo/blob counts
If half of your data (1.25GB) changes and it takes ~4 hours this means restic processes ~5.46GB/s. Not too bad, but it’s hard to tell if this is reasonable without knowing how much is changing exactly. If restic needs to re-chunk all of this files …
Files: 60802 new, 29365 changed, 144338 unmodified
Dirs: 0 new, 1 changed, 0 unmodified
Added to the repo: 49.283 GiB
processed 234505 files, 2.537 TiB in 27:08
snapshot f50b4cee saved
Something I should say wrong is “27:08”. I was not half an hour sure what takes on the backup. I left this working after 1 hour or so… and the day after I check it again.
Well… in fact 99% of the volume of the data didn’t changed (media files). What only changes was the small text files (sidecards)… and in this case only changes the timestamp…
Thanks for the additional information. I assume that the statistics you provided were from 0.9.6?
Some things I notice, some recommendations, and some more questions:
This backup you shared - was that a pretty normal 24-hour backup?
I noticed about 38% of files are either new or modified - about 90,000 files in total. That’s a lot, does that normally occur on a daily basis?
The total amount written to the repository was 49GB. That’s a fairly substantial amount of data - but as you noted, this is only about 2% of the total archive, and I wouldn’t expect it to take multiple hours.
I have not observed big differences between the reported duration and the actual duration, but I am interested. Can you time your backup more precisely to measure the difference between actual backup duration and reported backup duration? You can measure this actual duration easily using the time command like this: % time restic -r /Volumes/SOURCE_A backup /Volumes/SOURCE_B. This will report real, user and system time for the execution of the backup.
Can you share some statistics about your repository? (size of the repository in files and bytes? Number of snapshots?)
Here’s a command-line you can use to get more information about WHERE restic is spending its time:
This will create /tmp/my-bkup-output with double-verbose output and a timestamp on every line. From there, you can filter out the least interesting lines to gain insight into where restic is spending its time.