Slow speeds of Restic with Google Drive rclone backend

whereisaaron · July 19, 2018, 6:01pm

You could try copying your restic repo back and forth from Drive with rclone (no restic involved). That would test with the same file sizes. If you get the same slow speed then I would guess you are correct, it is the file sizes that limit restic speed.

Maybe there is (or could be) some option to increase the block sizes restic uses? If you did that you might get less de-duplication but maybe faster overall speed.

fd0 · July 19, 2018, 7:59pm

That’s not a good idea, this will increase the overhead, restic needs to manage all the blocks. A better idea is to increase the pack file size, which is currently hard coded.

Pneumaticat · July 19, 2018, 8:22pm

Testing as you suggested, I get the same half-line-speed:

2018/07/19 16:20:54 INFO  : data/3a/3a0760b85f7460add9d76a4b2166d66569521f49e37f4a246eaf649df39198fd: Deleted                                              
2018/07/19 16:20:54 INFO  :
Transferred:   849.887 MBytes (4.686 MBytes/s)
Errors:                 0
Checks:               152
Transferred:          152
Elapsed time:      3m1.3s
Transferring:
 *   ...63fa36d002cb4bbdbc862d05e910fdbc6eb303e1:  0% /5.162M, 0/s, -
 *   ...7a7be24d2e99fb997f23c1c9ff077574425296c1: 100% /5.545M, 4.248M/s, 0s
 *   ...8b4d23ca86fb05a69c696d01befd80523e97a266:  0% /4.825M, 0/s, -
 *   ...adca888e72b8e494e211d5f41f783359da7867b3: 100% /6.216M, 4.454M/s, 0s

So I think it is the small files after all.

I also get this error frequently, probably because of the small files creating a large number of requests to Google Drive:

2018/07/19 16:20:45 DEBUG : pacer: Rate limited, sleeping for 1.399764369s (1 consecutive low level retries)                                               
2018/07/19 16:20:45 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)                               
2018/07/19 16:20:45 DEBUG : pacer: Rate limited, sleeping for 2.281560672s (2 consecutive low level retries)                                               
2018/07/19 16:20:45 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)                               
2018/07/19 16:20:46 DEBUG : pacer: Rate limited, sleeping for 4.745774738s (3 consecutive low level retries)                                               
2018/07/19 16:20:46 DEBUG : pacer: low level retry 1/1 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)

Which is probably part of why it goes slowly.

@fd0, would increasing the pack file size be easy to do/make configurable? I think in this case it would definitely help.

fd0 · July 20, 2018, 6:28am

No, not easily. And I’m very reluctant to exposing this as a user-configurable option. If you want to experiment with it though, it’s easy to change in the source code here:

github.com

restic/restic/blob/67535e00a86469a0cebebd240f421b15199bbac2/internal/repository/packer_manager.go#L39


      
          }
          
          // packerManager keeps a list of open packs and creates new on demand.
          type packerManager struct {
          	be      Saver
          	key     *crypto.Key
          	pm      sync.Mutex
          	packers []*Packer
          }
          
          const minPackSize = 4 * 1024 * 1024
          
          // newPackerManager returns an new packer manager which writes temporary files
          // to a temporary directory
          func newPackerManager(be Saver, key *crypto.Key) *packerManager {
          	return &packerManager{
          		be:  be,
          		key: key,
          	}
          }

You can try setting minPackSize to something much larger than the default, like 64MiB (or even 256MiB), and try that. I think restic will cope just fine with larger packs, but I haven’t tried it in a long time. So please report back!

I think there’s no one-size-fits-all for packs, there are several trade-offs to decide and for now we go with a small pack size by default. In the long run I’d like to use a dynamic value which increases the pack size when restic detects that the backend connection has a high bandwidth.

Pneumaticat · July 20, 2018, 12:53pm

Changing the pack size to 256 MB does appear to have solved the speed problems! I get 6.9 MB/s:

rclone: Transferred:   7.327 GBytes (6.935 MBytes/s)Online Textbook-00000.warc.gz                                                                          
rclone: Errors:                 3ave of Dragonflies/The Cave of Dragonflies/hts-log.txt                                                                    
rclone: Checks:                 0
rclone: Transferred:           34
rclone: Elapsed time:     18m1.7s
rclone: Transferring:
rclone:  *   ...13a2312da250bbcfc6e0ef65dde3ba11be17de59: 90% /256.067M, 1.467M/s, 17s                                                                     
rclone:  *   ...33045875308bf9f2959929dc7ae3e921aa677b5d: 83% /257.002M, 1.056M/s, 40s                                                                     
rclone:  *   ...a59d07e9c1a59a85a73ae69fd5e0e1aecc76ed0e: 80% /256.629M, 1.095M/s, 46s                                                                     
rclone:  *   ...a6c7af6eb83384e68751762de7c1b45dba9db3c8: 90% /256.702M, 1.256M/s, 19s                                                                     
rclone:  *   ...de5261a00afa7a8612da9e45e0fa25dac5fe4c48: 73% /260.012M, 2.475M/s, 28s                                                                     
rclone:

Which I’m pretty sure is maxing out my current connection to Google Drive at this time (it tends to vary).

I haven’t noticed any other adverse effects of changing the pack size from my few minutes of usage; restic init and backup appear to work fine. I’ll report back again after using it for a little while longer.

Thank you for all your help, and your awesome work on restic!

whereisaaron · July 22, 2018, 7:54am

Looks like you are in part hitting against the pretty aggressive Google Drive rate limiting. Although it is not really a solution to this issue, you could consider using a B2 or S3 account for your backups. Those services simply charge you a few cents for high API rates, rather than slowing you down. You may be trying to leverage a Drive ‘unlimited’ storage plan. Even so, B2 storage is pretty cheap, and for less drama and fewer problems like this, it may be worth a couple dollars a month.

matt · July 22, 2018, 9:38pm

This is usually my recommendation too, it’s totally worth a few cents or dollars even if you have a free Google Drive. (“It’s a trap!”)

fd0 · July 23, 2018, 6:55am

FYI, I’ve got a beta account for B2 “application keys” (so you can create a second set of keys restricted to a particular bucket), I’m working on integrating this with restic.

Pneumaticat · July 24, 2018, 12:42am

Sounds pretty tempting, but I’ve got a lot of data to back up (~8 TB) with not that high of a budget to go with it I do agree that B2 would definitely be a better option if possible, though.

In any case, it seems that Google Drive is chugging along pretty well now – around 7 MB/s, which is honestly fine by me. Thank you to all!

durval · May 18, 2019, 3:11pm

Howdy everyone,

As mentioned here, I’m preparing to test this here – it’s kinda “life or death” as I really need to speed up my restic backup: it’s taking almost 24 hours to update just 24 hours of changed data, and I need to reduce it to 12 hours or less.

EDIT: not so much “life or death situation” anymore, as I managed to work around it by moving a large part of the backup to being updated only once per week, on Friday nights when it has the whole weekend to work. But I still would like it very much to speed this up.

@fd0 (or anyone with enough knowledge), can you please tell me:

Would a restic binary with a larger minPackSize be able to ‘interoperate’ on an already existing repository, ie read/write on it along with ‘standard’ (ie 4MiB PackSize) restic?
The way I understand it, packs are generated when blobs get written, so to realize the full benefit of upping minPackSize, the repository would need to be re-initialized and regenerated from the scratch, correct? Or would it be possible to somehow “repack” a current repository into a new, larger pack-sized one?

Also, @Pneumaticat, could you please give us an update? Are you still running restic with 256MB (or MiB, I presume) minPackSize? How is it working out for you? Have you tested restic restore? What about memory usage during backup/restore?

Thanks in advance,
– Durval.

cdhowie · May 18, 2019, 4:15pm

I believe so. The pack size is not fixed, rather there is a maximum. When building packs, restic will keep adding data until the size gets too big, then it will upload that pack and start building a new pack.

At the end of a backup, there’s usually not enough data left for a full pack and so a very small pack can be written with whatever is left.

All that to say: restic already needs to work with small and variable pack sizes, so changing this variable should not affect restic’s ability to work with existing packs.

My understanding is that uploads are throttled more heavily than downloads, so the existing smaller packs should not pose that much of a problem.

There is no way that I know of to rewrite everything. However, when you prune, any pack that contains an object that is no longer used will be repacked; if multiple packs contain an object that needs to be deleted, they will be combined. After several prune operations, you should see the average pack size in the repository get larger.

fd0 · May 19, 2019, 7:57am

Yes, there’s no other place in the source which requires a small pack size, restic will just take the files as they are stored in the backend. It’s a trade off so that people with tiny upstream bandwidth can use restic, and I have plans to adjust the pack size based on the backend upload speed. But that’s not an issue I can implement short term, sorry about that.

The reason why you also see larger files is that restic serialises metadata (file names, list of IDs of the content, modes, timestamps) as JSON and saves that as a tree blob to a file. If the directory is very large (number of files or size of files), the JSON document may grow much larger than 4MiB. Restic will still upload and process such a file.

Btw, the constant is called minPackSize because that’s the size a file has to reach before it can be considered “full enough” to be uploaded to the repo.

Some of the constraints I had in mind (from the top of my head):

The transfer of a single file should finish in reasonable time, even for users with low upstream bandwidth. Truncated files are of no use, only successfully uploaded files will be considered by e.g. rebuild-index. This takes low-end embedded systems with e.g. 512MiB into account (not considering the ongoing problems with the index loaded into memory completely).
A single file should fit into memory easily, so we can load a file during check and decrypt and verify all blobs stored in it. So a file size of e.g. 1GiB is probably too much
The file size should not be too small, after all one of the ideas of bundling together blobs into files is reducing the number of files stored in a backend.
When designing the repository format, I only had the local and sftp backend in mind and did not think about latency at all. It was simple back in the days

I hope this helps understanding the constraints a bit better

It’s a minimum, the minimal size a file must have to be uploaded to the backend. And you’re right about the last pack file being smaller (in some cases at least).

Correct.

durval · May 19, 2019, 3:40pm

Thanks for the detailed, thoughtful responses @cdhowie and @fd0.

I plan on doing some experimenting with changing minPackSize in the future. If/when I do, I will report back here.

Cheers,
– Durval.

bhansconnect · May 10, 2020, 7:10pm

Just wondering, is there any update on this?

Also, are there plans to expose these variables so you don’t need to recompile?

lrrrac · July 22, 2020, 11:46pm

Hey just curious about the current state of building from source. Ive been trying to build and the docker container has an issue with the flags github it uses. Ive been using the build.go to build the executible but changing the const doesnt seem to have an effect at all. Im working with a symmetric gigabit connection while restic maxes out at 100Mb upload btw.

Thanks!

rawtaz · November 1, 2021, 10:18pm

Before users came along

upsidedown · March 29, 2023, 5:35am

Sorry to re-open an old topic but I’m also getting veeery slow speeds on restoring and I’m increasing the pack size from now on and just stumbled on restic telling me that the limit is 128MB (I wanted to use 256), Is there a reason why such a limit? I thought we were free to such a size, the help (restic --help) and documentation doesn’t tell about any limit.

I’m currently getting 5-8 MiB/s download speeds so you can imagine the suffering.

rawtaz · March 29, 2023, 2:44pm

First, make sure you are running the latest restic version, 0.15.1 as of writing this. Does the problem persist?

upsidedown · March 29, 2023, 4:32pm

yes, using restic 0.15.1 compiled with go1.19.5 on windows/amd64, I’m somewhat confident is the pack size because using 128 made the backup reach max speed which if I remember correctly I’ve never seen restic reach before, I mean it wasn’t slow before by any means but I don’t remember restic reaching max speeds either in upload nor download.

rawtaz · March 29, 2023, 7:15pm

I haven’t read through it all but I know there was some reasoning around these things in https://github.com/restic/restic/pull/3731 in case you can find something of value there.