S3 backend: HTTP2 support

VShCReaJi22j4ifdym9Z · February 17, 2021, 12:38pm

Hello All,

I have been looking for ways to enable HTTP2 support when using restic 0.11.0 and restic 0.12.0 (mostly Windows) with the s3 backend. Particularly, the s3 backend is minio hosted on docker in the local network behind an nginx reverse proxy as per the official instructions at minio (both for nginx and minio). References: MinIO | Setup Nginx proxy with MinIO - Cookbook/Recipe .

In regards to what work and does not work:

I can see that minio client is able to use HTTP2 when talking to the same URL where the buckets are located
I can see that only restic is not doing HTTP2 via access logs on nginx
The previous link does not have configuration about HTTP2 and certificates, but I have added them to my instance of nginx and when I use google chrome it works as expected
I have read through the code and I believe that the only parameter to set for HTTP2 is done within the source code. There was a change just before 0.12.0, but it does not seem to apply to s3 backend. Ref: recent github commit HTTP2 HTTP backend
- note that s3 then uses the default http.Transport which includes support for HTTP2 by default
I have not used HTTP2 in this environment before, but there is some evidence that it has worked:
- Memory leak when backing up over http2 to minio behind Nginx · Issue #565 · restic/restic · GitHub
the certificate is valid. the issuer is letsencrypt via local certbot
I have only read through the code, I have not done any debugging as I am not familiar enough with the core of restic that then calls this part of the code.

Thanks,
V

fd0 · February 17, 2021, 3:00pm

Hm, that’s indeed odd. Recently, we had a pull request which added HTTP2 back (it needs to be enabled explicitly when configuring an HTTP transport), that’d be #3254. This commit is included in 0.12.0 but not in 0.11.0.

Within the source, I just verified that the s3 backend configures the HTTP transport passed to it here:

github.com

restic/restic/blob/27f241334e9245a212bc2aba4956a5c0392e5940/internal/backend/s3/s3.go#L75


			Client: &http.Client{
				Transport: http.DefaultTransport,
			},
		},
	})
	options := &minio.Options{
		Creds:     creds,
		Secure:    !cfg.UseHTTP,
		Region:    cfg.Region,
		Transport: rt,
	}
	switch strings.ToLower(cfg.BucketLookup) {
	case "", "auto":
		options.BucketLookup = minio.BucketLookupAuto
	case "dns":
		options.BucketLookup = minio.BucketLookupDNS
	case "path":
		options.BucketLookup = minio.BucketLookupPath
	default:

It gets called here:

github.com

restic/restic/blob/27f241334e9245a212bc2aba4956a5c0392e5940/cmd/restic/global.go#L701


	// wrap the transport so that the throughput via HTTP is limited
	lim := limiter.NewStaticLimiter(gopts.LimitUploadKb, gopts.LimitDownloadKb)
	rt = lim.Transport(rt)
	switch loc.Scheme {
	case "local":
		be, err = local.Open(globalOptions.ctx, cfg.(local.Config))
	case "sftp":
		be, err = sftp.Open(globalOptions.ctx, cfg.(sftp.Config))
	case "s3":
		be, err = s3.Open(globalOptions.ctx, cfg.(s3.Config), rt)
	case "gs":
		be, err = gs.Open(cfg.(gs.Config), rt)
	case "azure":
		be, err = azure.Open(cfg.(azure.Config), rt)
	case "swift":
		be, err = swift.Open(cfg.(swift.Config), rt)
	case "b2":
		be, err = b2.Open(globalOptions.ctx, cfg.(b2.Config), rt)
	case "rest":
		be, err = rest.Open(cfg.(rest.Config), rt)

And the rt is instanciated here:

github.com

restic/restic/blob/27f241334e9245a212bc2aba4956a5c0392e5940/cmd/restic/global.go#L686


	cfg, err := parseConfig(loc, opts)
	if err != nil {
		return nil, err
	}
	tropts := backend.TransportOptions{
		RootCertFilenames:        globalOptions.CACerts,
		TLSClientCertKeyFilename: globalOptions.TLSClientCert,
	}
	rt, err := backend.Transport(tropts)
	if err != nil {
		return nil, err
	}
	// wrap the transport so that the throughput via HTTP is limited
	lim := limiter.NewStaticLimiter(gopts.LimitUploadKb, gopts.LimitDownloadKb)
	rt = lim.Transport(rt)
	switch loc.Scheme {
	case "local":

which calls Transport() from here:

github.com

restic/restic/blob/27f241334e9245a212bc2aba4956a5c0392e5940/internal/backend/http_transport.go#L64-L80


func Transport(opts TransportOptions) (http.RoundTripper, error) {
	// copied from net/http
	tr := &http.Transport{
		Proxy: http.ProxyFromEnvironment,
		DialContext: (&net.Dialer{
			Timeout:   30 * time.Second,
			KeepAlive: 30 * time.Second,
			DualStack: true,
		}).DialContext,
		ForceAttemptHTTP2:     true,
		MaxIdleConns:          100,
		MaxIdleConnsPerHost:   100,
		IdleConnTimeout:       90 * time.Second,
		TLSHandshakeTimeout:   10 * time.Second,
		ExpectContinueTimeout: 1 * time.Second,
		TLSClientConfig:       &tls.Config{},
	}

And there you can see that ForceAttemptHTTP2 is indeed set to true. So it should work…

Can you please confirm that it does not work with restic 0.12.0? If that’s the case, please open an issue on GitHub, then you’ve found a bug

VShCReaJi22j4ifdym9Z · February 18, 2021, 3:30pm

fd0,

Thank you for this. I was about to dissect the code, but after sleeping on this issue for a couple of nights. I realized that I was using a different reverse proxy at the time I realized I had issues using HTTP2. I did entirely miss the first 3-4 times I reviewed this issue, because I was using s different host and server pair at the time.

I have now made sure that the temporary reverse proxy I was using is also set to to use ‘http2’ in the listen directive of nginx configuration.

Thanks,
V

fd0 · February 18, 2021, 7:48pm

Thanks for the feedback and explaining the cause of your issue. Sometimes it’s easy in hindsight (and having a good night of sleep is a remarkable useful debug tool).