Speeding up backing up of many files

Getting Help
1

Hello!

I’ve just learned about restic and I’m trying to backup around 2 TB of pictures/videos, most of them being relatively small but some of them are pretty big files for very long videos. After trying to directly back up everything to my Google Drive account using rclone and seeing that I had only uploaded around 10% after 2 days, I decided I’d look into another solution and found out about restic. Unfortunately, restic seems to be even slower with an ETA of 500 hours (~20 days) for 1.7 TB…

Some information:

  • Uploading to an encrypted rclone remote on Google Drive
  • Using Windows
  • Bandwidth isn’t the problem, I have gigabit upload
  • The files are stored on an HDD but I doubt that’s the limiting factor here
  • I have around 1.2 M files

Is there anything I can do to speed up the process? Or anything I’m doing wrong?

Thanks!

2

My understanding is that Google Drive heavily throttles individual file operations. Restic targets 8MB for each pack file, so divide 2TB by 8MB = ~260 thousand packs. Uploading hundreds of thousands of files is going to take awhile because of the per-file request throttling.

Google Drive is not well-suited for use with restic, in my opinion.

1 Like
3

gigabit doesn’t mean much in the real world. I have gigabit up/down and the only time I see this is testing a nearby server. I use wasabi for cloud storage and they provide a speed test through their management console. At the moment their tests shows 857 Mbps down / 213 Mbps up. That’s from east coast Canada to Manassas VA.

4

Google Drive throttles file operations to 2 uploads per second. Given the number of packs you’ve calculated, that should take around 130 000 seconds (assuming instant upload) which is a little more than a day and a half (~36 hours). The current restic estimation is now 330 hours - so around 10 times more, which means I’m barely uploading two files every 10 seconds…
After monitoring the upload speed, it’s struggling to reach even 1 MB/s so I assume they’re also throttling the speed of individual file operations…

I just found out about this: Slow speeds of Restic with Google Drive rclone backend - #7 by fd0 so I’ll try it out, but I’m wondering if there is a technical reason for keeping the pack to such a small - fixed - size given that it’s been working properly for others?

I understand that I’m likely never going to max out my upload bandwidth, I was just pointing out that the limiting factor isn’t my connection.

5

See this open PR for the technical details about changing it:

1 Like
6

Thanks! Compiled restic from that PR and increased the pack size to 256 MB, went from a 500 hours ETA to 150 hours. I’m now uploading to around 4-8 MB/s. One thing I’ve noticed though is that it seems restic is only processing/creating one pack at a time, and therefore only uploads one pack at a time.
I haven’t found anything in the docs regarding parallelization of those steps, is it at all possible? It’d essentially divide the time by two again by leveraging the maximum number of concurrent uploads to Google Drive.