Slow Backup to hosted S3 with minimal cpu ,io and network loads

Command : restic --read-concurrency=4 -o s3.connections=6 --pack-size=60 backup /mnt/alldrives/3dfilesdxb
Host: truenas / Restic version restic 0.18.0

This is backing up to my hosted minio S3 server. with or without s3.connections parameter, there is only 1 connection taking place maxing out at 2.5MB/sec. On the host (where restic command is run, the cpu loads is quite low. I am unsure why the system wont max out.

The files range from 5mb to 1-2GB in size, dataset is around 2TB

I have ample of bandwidth on the hosted instance and cpu as well.

OK after breaking my head for over 5 hours, i finally found it; for anyone else out there DON’T USE HTTP2. h2 multiplexing reduces it to 1 single connection.

1 Like

Interesting. How do you enable or disable http2?

Depending on your setup, I was using nginx proxy. Maybe this problem won’t happen on haproxy, not sure. But in nginx it’s just about removing http2 from your listen or http2 enable parameter. But as I said depends on what you’re using using YMMV

Oh ok, you’re talking about server side. I thought you’d found a client side switch or something. I assumed you’d go straight to minio, but I’ve never used it so I’m not sure if the details. I use AWS S3 as my main target, to get data offsite.

http2 should allow multiple streams. Maybe it’s a limitation of Restic, or how Nginx works with Restic, or the configuration you had. Sometimes there’s some subtle things in the Nginx configuration that need tweaking, not for http2 specifically, I just remember taking some time configurating websockets to work properly recently.

I don’t think there is web sockets involved in restic to speak with any s3 provider. Yes it’s an nginx thing where it is multiplexing and merging the connections into a single connection. From what I understand haproxy shouldn’t have this issue, but I’ve never set it up. That said, unless there’s something I haven’t taken into account, is there anything else I would be losing out on if I do continue down with h2 ? TLS 1.2 remains as is, there is the additional headers overhead, but that’s all I can think of ATM by avoiding h2 .

Websockets was just an example of having to tweaks Nginx quite a bit to get things working.