Should we discourage users from using Google Drive?

Features and Ideas
19

Here is a quick example showing that increasing the blob size from 10MB to 100MB might give us a 4x speedup and increasing the chunk size from 8M to 32M another 20%.

I left the default --transfers as 4 so rclone uploads that many files at once. When using rclone serve it is up to restic how many transfers are done at once and I’m not sure what the answer to that is.

(I did part of a test with 250M chunks which was running at abou 75MB/s but I’ve filled the quota of my drive and even though I’ve deleted stuff Google won’t let me upload anything at the moment!)

10 MB files

$ rclone size 1GB-of-10MB-files
Total objects: 100
Total size: 1000 MBytes (1048576000 Bytes)

default chunk size 8M

$ rclone sync -P 1GB-of-10MB-files TestDrive:1GB-of-10MB-files
Transferred:         1000M / 1000 MBytes, 100%, 15.537 MBytes/s, ETA 0s
Errors:                 0
Checks:                 0 / 0, -
Transferred:          100 / 100, 100%
Elapsed time:      1m4.3s

32M chunk size

$ rclone sync --drive-chunk-size 32M -P 1GB-of-10MB-files TestDrive:1GB-of-10MB-files-2
Transferred:         1000M / 1000 MBytes, 100%, 17.058 MBytes/s, ETA 0s
Errors:                 0
Checks:                 0 / 0, -
Transferred:          100 / 100, 100%
Elapsed time:       58.6s

100 MB files

$ rclone size 1GB-of-100MB-files
Total objects: 10
Total size: 1000 MBytes (1048576000 Bytes)

default chunk size 8M

$ rclone sync -P 1GB-of-100MB-files TestDrive:1GB-of-100MB-files
Transferred:         1000M / 1000 MBytes, 100%, 51.124 MBytes/s, ETA 0s
Errors:                 0
Checks:                 0 / 0, -
Transferred:           10 / 10, 100%
Elapsed time:       19.5s

chunk size 32M

$ rclone sync --drive-chunk-size 32M -P 1GB-of-100MB-files TestDrive:1GB-of-100MB-files-2
Transferred:         1000M / 1000 MBytes, 100%, 60.730 MBytes/s, ETA 0s
Errors:                 0
Checks:                 0 / 0, -
Transferred:           10 / 10, 100%
Elapsed time:       16.4s

BTW I discovered a post where @fd0 describes how to increase the pack size, so if you want to experiment with something that would be the thing!

2 Likes