Restic hangs on Google Cloud Storage

This problem started several months ago. Unfortunately, I don’t remember if it corresponds with an OS update or a restic upgrade. When I run the following command for an existing backup, it gets most of the way through, then hangs at 99%+ on a recently modified file:

restic backup -vv --host server \
  --compression max \
  --exclude-larger-than 500G \
  -r gs:my-repo:/ /tank

My setup is a little weird. I’m backing up a zfs snapshot over nfs, but because I don’t want it to appear to be a snapshot, I rewrite the paths with bind mounts and chroot.

This is my env. I set the GODEBUG flags because I read about an issue with cifs and was paranoid it also applied to nfs, and I tried enabling and disabling http2.

export RESTIC_PASSWORD=...
export GOOGLE_PROJECT_ID=...
export GOOGLE_APPLICATION_CREDENTIALS=...
export GODEBUG=http2client=0,asyncpreemptoff=1,http2debug=1

I’m running this from FreeBSD, so I tried both a ports build and an official restic build with Linux emulation:

$ restic version
restic 0.17.1 compiled with go1.21.13 on freebsd/amd64

$ ./restic_0.17.1_linux_amd64 version
restic 0.17.1 compiled with go1.23.1 on linux/amd64

My internet connection is 300Mbps fiber, so I doubt that’s the issue.

As part of debugging, I wanted to see what was going on with the data, so I ran traffic through mitmproxy on a Linux host. Strangely, it fixed my problem.

export https_proxy="http://linux-host:12345"
--cacert ~/.mitmproxy/mitmproxy-ca.pem

I’m not sure where to look next. My only theories are throttling by Google and a bug in the FreeBSD network stack.

A few things you could try:

set the DEBUG_LOG=restic.log environment variable to let restic write a log file. That might show what it is doing.

Or send a SIGQUIT signal to restic. That causes the Go runtime to terminate and print a stacktrace. Restic is designed to gracefully handle termination at arbitrary points in time. The only gotcha is that it will leave a lock file behind, which you can remove using restic unlock.