Speeding up backing up of many files

[Hawpe]

@cdhowie

My understanding is that Google Drive heavily throttles individual file operations. Restic targets 8MB for each pack file, so divide 2TB by 8MB = ~260 thousand packs. Uploading hundreds of thousands of files is going to take awhile because of the per-file request throttling.

Google Drive is not well-suited for use with restic, in my opinion.

Google Drive throttles file operations to 2 uploads per second. Given the number of packs you’ve calculated, that should take around 130 000 seconds (assuming instant upload) which is a little more than a day and a half (~36 hours). The current restic estimation is now 330 hours - so around 10 times more, which means I’m barely uploading two files every 10 seconds…
After monitoring the upload speed, it’s struggling to reach even 1 MB/s so I assume they’re also throttling the speed of individual file operations…

I just found out about this: Slow speeds of Restic with Google Drive rclone backend - #7 by fd0 so I’ll try it out, but I’m wondering if there is a technical reason for keeping the pack to such a small - fixed - size given that it’s been working properly for others?

@doscott

gigabit doesn’t mean much in the real world. I have gigabit up/down and the only time I see this is testing a nearby server. I use wasabi for cloud storage and they provide a speed test through their management console. At the moment their tests shows 857 Mbps down / 213 Mbps up. That’s from east coast Canada to Manassas VA.

I understand that I’m likely never going to max out my upload bandwidth, I was just pointing out that the limiting factor isn’t my connection.