B2 vs GCS speed expectations

[gened]

Heya everyone,

I’m hoping for a bit of a sanity check. I’ve been using B2 for a few months to wrangle about 10TB of revolving data [more data is added as some ages out]. Typically, our upstream transfer hovers around 10MB/s. As our data grows, we’re looking for a faster service and have recently trialed a GCS nearline bucket in a DC close-ish to our location.

The transfer rate i’m seeing with gsutil (google’s tool) hovers around 100MB/s, while using restic the rate hovers around 10MB/s. I found this a bit surprising. Any insight would be greatly appreciated.

I’ve tried various iterations of the gs.connections=N param, but that did not really seem to make any difference.

Best,

Gene

[cdhowie]

Restic only uses two worker threads, which is likely the reason for the discrepancy you’re seeing. Setting gs.connections above 2 will have no effect.

[gened]

Thanks for the input,

Much appreciated. Is this likely to change? Or does requiring more throughput preclude me from using restic?

Best,

G

[cdhowie]

This thread might be helpful, as it documents the restriction and the future plans to solve it.

-o b2.connections=N seems to have no impact

Hi, (latest restic: 0.9.3, on debian 9, sufficient RAM and disk space) as I’m in EU and ping delay to backblaze is about 180ms, I need concurrent connections. Having tried settings of 10, 50, 100 and 500, I see no difference: the number of connections really open to backblaze (as reported by netstat -an | grep EST | grep 443 and filtering out the non-backblaze connections), is mostly 3, and sometimes goes to 5. Irrespective of -o b2.connections=... Also (but that is not the issue, my questio…

[gened]

Thanks cdhowie.

I look forward to this limitation being lifted one day.