Restic hangs on Google Cloud Storage

Getting Help
1

This problem started several months ago. Unfortunately, I don’t remember if it corresponds with an OS update or a restic upgrade. When I run the following command for an existing backup, it gets most of the way through, then hangs at 99%+ on a recently modified file:

restic backup -vv --host server \
  --compression max \
  --exclude-larger-than 500G \
  -r gs:my-repo:/ /tank

My setup is a little weird. I’m backing up a zfs snapshot over nfs, but because I don’t want it to appear to be a snapshot, I rewrite the paths with bind mounts and chroot.

This is my env. I set the GODEBUG flags because I read about an issue with cifs and was paranoid it also applied to nfs, and I tried enabling and disabling http2.

export RESTIC_PASSWORD=...
export GOOGLE_PROJECT_ID=...
export GOOGLE_APPLICATION_CREDENTIALS=...
export GODEBUG=http2client=0,asyncpreemptoff=1,http2debug=1

I’m running this from FreeBSD, so I tried both a ports build and an official restic build with Linux emulation:

$ restic version
restic 0.17.1 compiled with go1.21.13 on freebsd/amd64

$ ./restic_0.17.1_linux_amd64 version
restic 0.17.1 compiled with go1.23.1 on linux/amd64

My internet connection is 300Mbps fiber, so I doubt that’s the issue.

As part of debugging, I wanted to see what was going on with the data, so I ran traffic through mitmproxy on a Linux host. Strangely, it fixed my problem.

export https_proxy="http://linux-host:12345"
--cacert ~/.mitmproxy/mitmproxy-ca.pem

I’m not sure where to look next. My only theories are throttling by Google and a bug in the FreeBSD network stack.

2

A few things you could try:

set the DEBUG_LOG=restic.log environment variable to let restic write a log file. That might show what it is doing.

Or send a SIGQUIT signal to restic. That causes the Go runtime to terminate and print a stacktrace. Restic is designed to gracefully handle termination at arbitrary points in time. The only gotcha is that it will leave a lock file behind, which you can remove using restic unlock.

3

I finally got around to trying this. I updated Restic from FreeBSD’s ports. I also saw issues with the Linux release binary running under FreeBSD’s Linux compatibility, but I wanted to keep it simple for this test.

restic 0.17.3 compiled with go1.21.13 on freebsd/amd64

I ran a backup with pretty simple options and env:

export DEBUG_LOG=/root/restic3.log
restic backup -v \
  --compression max \
  --exclude-larger-than 500G \
  -r "$REPO" /tank

Here’s some of the log when it got stuck. It’s on a GCS delete that took almost a minute. There was nothing else in the log when it happened.

2025/01/02 20:01:54 debug/round_tripper.go:93   debug.loggingRoundTripper.RoundTrip     130     ------------  HTTP REQUEST -----------
DELETE /storage/v1/b/sunshineorigami/o/locks%2F1d1f521ecd40eb96bcdffed77217503b3de7bfbd5e34429bee286e3ef96555ff?alt=json&prettyPrint=false HTTP/1.1
Host: storage.googleapis.com
User-Agent: google-api-go-client/0.5
Authorization: **redacted**
X-Goog-Api-Client: gl-go/1.21.13 gdcl/0.187.0 gccl-invocation-id/acb1dd43-0a3b-4a1b-b500-7f52c8487cf7 gccl-attempt-count/1 gl-go/1.21.13 gccl/1.43.0
X-Goog-Gcs-Idempotency-Token: acb1dd43-0a3b-4a1b-b500-7f52c8487cf7
Accept-Encoding: gzip


2025/01/02 20:02:49 debug/round_tripper.go:110  debug.loggingRoundTripper.RoundTrip     130     ------------  HTTP RESPONSE ----------
HTTP/2.0 204 No Content
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Content-Type: application/json
Date: Fri, 03 Jan 2025 04:04:58 GMT
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Pragma: no-cache
Server: UploadServer
Vary: Origin
Vary: X-Origin
X-Guploader-Uploadid: AFiumC4WXgs-R3kEvdOxKSNKCIS9cTmP_qAdlKu6yq3z7ms1-FyJTLsMg0Iv5D7Mj7AHId-A

The next time it happened, I sent a SIGQUIT. Here are all the call stacks: https://pastebin.com/h3BU7Ncy

4

I appreciate your efforts in updating Restic via FreeBSD’s ports. It’s common to encounter issues with Linux binaries when running them under FreeBSD’s Linux compatibility layer. Keeping it simple for your test is a good approach.If you face any specific errors, such as those related to FUSE, ensure that the necessary kernel modules are loaded. For example, running kldload fuse or adding fuse_load="YES" to /boot/loader.conf can resolve FUSE-related issues when trying to mount repositories. Additionally, check if all required dependencies are installed, as missing packages can lead to functionality problems

5

The backup command does not use FUSE, so that’s irrelevant here.

Based on a quick glance that looks like the HTTP2 connection got stuck somehow. Please try building restic with Go 1.23 and check whether that makes a difference.

6

Should I hack around with ports to get it to build with Go 1.23 (it uses 1.21 right now), or should I use a pre-built release binary from Releases · restic/restic · GitHub?

7

I’d just start with the official release binary to see whether is solves the problem. If it does, you still try to get ports to use the latest go version. (Go 1.21 is also problematic insofar as it no longer receives security updates).