I have recently started using Restic for backing up data from a few remote servers to an offsite location; and so far; it ss been great in terms of reliability and security. Although; I am running into some issues with backup speed; especially when dealing with slower or unreliable network connections.
The servers I am backing up have relatively large datasets around 500 GB each; and while the initial backup took a while; the incremental backups are still slower than I could hoped. I am aware that Restic deduplicates and only transfers new data; but I am still seeing long transfer times; particularly when the connection quality fluctuates.
Should I be setting this up to handle inconsistent network speeds better? Would tweaking the chunk size help in any way with performance, and how would I go about doing that safely?
Does enabling compression before backing up make a difference in speed; or would it only impact storage costs? Would running more frequent; smaller backups improve the overall performance?
I would not expect much difference here but I guess it depends somewhat on the file sizes you’re backing up.
Again, depending on the files you’re backing up, compession does make a difference. I always have it turned on (which restic defaults to anyway). All in all more frequent backups should be slower if anything, although each backup run could be faster, of course.
Question: what’s the problem with the backups taking longer? You won’t find magic solutions transferring lots of data via slow connections but if the backup window is too short, you might consider doing a local backup and then rsyncing that to the remote location afterwards when the servers are busy doing other things. If the servers’ RAM and CPU isn’t the problem you might do it locally on the server or otherwise to a local backup server and then on to the remote location.
Please provide a few more details. In particular, which backend do you use? How slow is the connection, which latency? What does connection quality fluctuations mean? Varying bandwidth? Massive packet loss, etc?
What is “this”? The chunk size in restic cannot be customized, but it wouldn’t have any effect only upload speed either. You probably mean “pack size”. For very slow connections, it could make sense to reduce it to 4 or 8 MB. But you’ll have to try out whether that results in an improvement or not.
Compression typically reduces how much data has to be uploaded. For slow connections that can make a drastic speed difference. What do you mean with “enabling compression before backing up”? Restic already provides builtin compression, which is enabled by default for repository format version 2 repos.
Would tweaking the chunk size help in any way with performance, and how would I go about doing that safely? 