Estimating backend storage requirements

[cfbao]

@HeavyThumper

I’m assuming the primary limiter is rclone - that because restic can’t talk to the backend directly there’s a significant level of inefficiency.

No. Delays caused by communication between two local programs is unnoticeable, and completely negligible when compared to the network latency.

@HeavyThumper

Is there a way of estimating the transactions/bandwidth for upload/download that can be used with the various backend providers’ calculators to estimate storage costs?

Not currently supported by restic (if possible at all).
One way to increase speed and reduce number of transactions is to increase pack size. This isn’t a currently supported feature either, but some users have experimented tweaking restic source code to use larger pack size, and with promising results.

Relavant discussions and user experiments:

Slow speeds of Restic with Google Drive rclone backend

Changing the pack size to 256 MB does appear to have solved the speed problems! I get 6.9 MB/s: rclone: Transferred: 7.327 GBytes (6.935 MBytes/s)Online Textbook-00000.warc.gz rclone: Errors: 3ave of Dragonflies/The Cave of Dragonflies/hts-log.txt rclone: Checks: 0 rclone: Transferred: 34 rclone: Elapsed time: …

How to set the Min/Max Pack Size?

I modified the Pack Size to be about 250MB (most of my files are immutable and are between 100MB-2GB in this repo) and have been using it for the last month. I’ve stored about 4.7TB with it and haven’t had any issues so far doing multiple backup, check, prune or restore commands. I did note that using ‘mount’ crashed out every time but it may not be anything to do with the Pack Size, I haven’t tried mount on the official binary. I haven’t tried to restore using the official binary yet to see wh…

Should we discourage users from using Google Drive?

If I look at my own backups I see an average pack size of 4.7MiB. The largest ones are about 11MiB. I’d argue that this is too small for any cloud storage system as the round trip times will really be eating into the upload bandwidth. Uploading multiple packs at once will help with this of course. However smaller packs also cost more transactions which cost actual money on some storage platforms (eg s3, b2). So an option to increase the pack size would increase transfer speeds on remote …

Control the minimal pack files size

I have a huge amount of files (~100 millions and total size ~50Tb), which should be backed up in Openstack Swift. Unfortunately Swift doesn’t recommend to store more than 1 million files in one container. I was hoping that restic can pack multiple files into one pack and it actually can, but according to the documentation the maximum pack file size is 8 MiB (my tests showed me 4 MiB and actually here is why: https://github.com/restic/restic/blob/2aa6b496519ef65c5cedc02aaaf2f3495137e6a5/internal…