I’ve just learned about restic and I’m trying to backup around 2 TB of pictures/videos, most of them being relatively small but some of them are pretty big files for very long videos. After trying to directly back up everything to my Google Drive account using rclone and seeing that I had only uploaded around 10% after 2 days, I decided I’d look into another solution and found out about restic. Unfortunately, restic seems to be even slower with an ETA of 500 hours (~20 days) for 1.7 TB…
Uploading to an encrypted rclone remote on Google Drive
Bandwidth isn’t the problem, I have gigabit upload
The files are stored on an HDD but I doubt that’s the limiting factor here
I have around 1.2 M files
Is there anything I can do to speed up the process? Or anything I’m doing wrong?
My understanding is that Google Drive heavily throttles individual file operations. Restic targets 8MB for each pack file, so divide 2TB by 8MB = ~260 thousand packs. Uploading hundreds of thousands of files is going to take awhile because of the per-file request throttling.
Google Drive is not well-suited for use with restic, in my opinion.
gigabit doesn’t mean much in the real world. I have gigabit up/down and the only time I see this is testing a nearby server. I use wasabi for cloud storage and they provide a speed test through their management console. At the moment their tests shows 857 Mbps down / 213 Mbps up. That’s from east coast Canada to Manassas VA.
Google Drive throttles file operations to 2 uploads per second. Given the number of packs you’ve calculated, that should take around 130 000 seconds (assuming instant upload) which is a little more than a day and a half (~36 hours). The current restic estimation is now 330 hours - so around 10 times more, which means I’m barely uploading two files every 10 seconds…
After monitoring the upload speed, it’s struggling to reach even 1 MB/s so I assume they’re also throttling the speed of individual file operations…
Thanks! Compiled restic from that PR and increased the pack size to 256 MB, went from a 500 hours ETA to 150 hours. I’m now uploading to around 4-8 MB/s. One thing I’ve noticed though is that it seems restic is only processing/creating one pack at a time, and therefore only uploads one pack at a time.
I haven’t found anything in the docs regarding parallelization of those steps, is it at all possible? It’d essentially divide the time by two again by leveraging the maximum number of concurrent uploads to Google Drive.