B2 vs GCS speed expectations

Heya everyone,

I’m hoping for a bit of a sanity check. I’ve been using B2 for a few months to wrangle about 10TB of revolving data [more data is added as some ages out]. Typically, our upstream transfer hovers around 10MB/s. As our data grows, we’re looking for a faster service and have recently trialed a GCS nearline bucket in a DC close-ish to our location.

The transfer rate i’m seeing with gsutil (google’s tool) hovers around 100MB/s, while using restic the rate hovers around 10MB/s. I found this a bit surprising. Any insight would be greatly appreciated.

I’ve tried various iterations of the gs.connections=N param, but that did not really seem to make any difference.

Best,

Gene

Restic only uses two worker threads, which is likely the reason for the discrepancy you’re seeing. Setting gs.connections above 2 will have no effect.

Thanks for the input,

Much appreciated. Is this likely to change? Or does requiring more throughput preclude me from using restic?

Best,

G

This thread might be helpful, as it documents the restriction and the future plans to solve it.

Thanks cdhowie.

I look forward to this limitation being lifted one day.