No output when backing up to Backblaze B2

I’ve been using restic to backup files to my Backblaze B2 bucket via an automated weekly cron job for about a couple of years now. Upon checking a short while ago though, it looks like no backups have been created since July 2019.

When I manually run “restic snapshots”, I get no textual output on the terminal, even after leaving it for a few days. I just tried to create a new B2 bucket and initialize a new restic repository in it, but I run into the same issue. Creating a local repository seems to work though. I tried changing my B2 master key and updating the B2_ACCOUNT_KEY environment variable, but still the same issue. I was using restic 0.8.0 on Linux Mint eariler, so I tried testing with restic 0.9.6 and still the same issue.

Any ideas what might be going wrong are much appreciated.

Please take a look directly at the original B2 bucket. Every restic repository has a snaphots directory which contains 1 file per snapshot in your repository. Anything in there?

The newest file in the snapshots directory of the original bucket was created in July 2019. The newly created bucket is empty despite trying to initialize a new repository in it.

If you run a backup manually, does the output contain any errors?

Again, the command hangs with no textual response. Screenshot attached.

Can you run with -vv?

It gets stuck at this point:

% ~/bin/restic backup --cleanup-cache --files-from=include_uc.txt --exclude-file=exclude_uc.tx -vv
open repository

Hmm. Is the process doing any network I/O when it gets stuck? Can you kill it with SIGABRT and paste the resulting stack dump?

Sorry, am not that experienced with this type of troubleshooting. Could you please advise how I might go about getting this information.

Assuming you have no other restic processes running, try pkill -6 restic. If that doesn’t work, use ps aux | grep restic to find the PID of the restic process, then run kill -6 THAT_PID. You can do this in a separate terminal than the one running restic.

Thanks a lot @rawtaz. Here’s the output after killing the process using pkill -6 restic:

 % ~/bin/restic backup --cleanup-cache --files-from=include_uc.txt --exclude-file=exclude_uc.tx -vv
open repository
SIGABRT: abort
PC=0x45d1f3 m=0 sigcode=0

goroutine 19 [syscall]:
runtime.notetsleepg(0x14bdbc0, 0xb9d18986, 0x0)
	/usr/local/go/src/runtime/lock_futex.go:227 +0x34 fp=0xc000053760 sp=0xc000053730 pc=0x40aba4
runtime.timerproc(0x14bdba0)
	/usr/local/go/src/runtime/time.go:311 +0x2f1 fp=0xc0000537d8 sp=0xc000053760 pc=0x44c0c1
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0000537e0 sp=0xc0000537d8 pc=0x45b2c1
created by runtime.(*timersBucket).addtimerLocked
	/usr/local/go/src/runtime/time.go:169 +0x10e

goroutine 1 [select]:
net/http.(*Transport).getConn(0xc000388000, 0xc0003d60c0, 0x0, 0xc0004f4e80, 0x5, 0xc0004cc7c0, 0x18, 0x0, 0x0, 0x0, ...)
	/usr/local/go/src/net/http/transport.go:1234 +0x553
net/http.(*Transport).roundTrip(0xc000388000, 0xc000229800, 0xc0002f0110, 0xc0002f017b, 0x1)
	/usr/local/go/src/net/http/transport.go:522 +0x603
net/http.(*Transport).RoundTrip(0xc000388000, 0xc000229800, 0xc0005e1ec0, 0x4e7a54, 0xc0002f0110)
	/usr/local/go/src/net/http/roundtrip.go:17 +0x35
github.com/restic/restic/internal/limiter.staticLimiter.roundTripper(0x0, 0x0, 0xf0d520, 0xc000388000, 0xc000229800, 0x762a63afc288f, 0x1344dcc4, 0x1344dcc4005e1f70)
	/restic/internal/limiter/static_limiter.go:63 +0x67
github.com/restic/restic/internal/limiter.staticLimiter.Transport.func1(0xc000229800, 0x5e3de98e, 0x1344dcc4, 0x762a63afc288f)
	/restic/internal/limiter/static_limiter.go:78 +0x4f
github.com/restic/restic/internal/limiter.roundTripper.RoundTrip(0xc0002187e0, 0xc000229800, 0x14b9140, 0xdda5bd, 0x18)
	/restic/internal/limiter/static_limiter.go:52 +0x30
github.com/kurin/blazer/b2.(*clientTransport).RoundTrip(0xc0002200e0, 0xc000229800, 0xc532c0, 0x9, 0x9)
	/restic/vendor/github.com/kurin/blazer/b2/b2.go:159 +0xe5
github.com/kurin/blazer/base.makeNetRequest(0xf20fe0, 0xc00013d8c0, 0xc000229700, 0xf0c5a0, 0xc0002200e0, 0xc0005d4a50, 0x2, 0x0)
	/restic/vendor/github.com/kurin/blazer/base/base.go:292 +0xfc
github.com/kurin/blazer/base.(*Bucket).DownloadFileByName(0xc000137e00, 0xf20fe0, 0xc00013d8c0, 0xdca1b6, 0x6, 0x0, 0x1, 0x0, 0x0, 0x0)
	/restic/vendor/github.com/kurin/blazer/base/base.go:1120 +0x62e
github.com/kurin/blazer/b2.(*b2Bucket).downloadFileByName(0xc00021a1a8, 0xf20fe0, 0xc00013d8c0, 0xdca1b6, 0x6, 0x0, 0x1, 0xc0005e2620, 0x2, 0x2, ...)
	/restic/vendor/github.com/kurin/blazer/b2/baseline.go:371 +0x91
github.com/kurin/blazer/b2.(*beBucket).downloadFileByName.func1.1(0x1, 0x1)
	/restic/vendor/github.com/kurin/blazer/b2/backend.go:434 +0x7a
github.com/kurin/blazer/b2.withReauth(0xf20fe0, 0xc00013d8c0, 0xf2cb00, 0xc000380090, 0xc0005e2578, 0xc0000302a0, 0xc000000208)
	/restic/vendor/github.com/kurin/blazer/b2/backend.go:766 +0x2b
github.com/kurin/blazer/b2.(*beBucket).downloadFileByName.func1(0xc0005e2620, 0xc0005e2608)
	/restic/vendor/github.com/kurin/blazer/b2/backend.go:444 +0xcd
github.com/kurin/blazer/b2.withBackoff(0xf20fe0, 0xc00013d8c0, 0xf2cb00, 0xc000380090, 0xc0005e26c8, 0x0, 0x0)
	/restic/vendor/github.com/kurin/blazer/b2/backend.go:747 +0x4d
github.com/kurin/blazer/b2.(*beBucket).downloadFileByName(0xc0005d28e0, 0xf20fe0, 0xc00013d8c0, 0xdca1b6, 0x6, 0x0, 0x1, 0xc0005e27c0, 0x40e7fb, 0xc000380088, ...)
	/restic/vendor/github.com/kurin/blazer/b2/backend.go:446 +0xf8
github.com/kurin/blazer/b2.(*Bucket).getObject(0xc0004692f0, 0xf20fe0, 0xc00013d8c0, 0xdca1b6, 0x6, 0xc82500, 0xc000218750, 0xc0005e2800)
	/restic/vendor/github.com/kurin/blazer/b2/b2.go:616 +0x80
github.com/kurin/blazer/b2.(*Object).ensure(0xc0005e29a0, 0xf20fe0, 0xc00013d8c0, 0x20, 0x20)
	/restic/vendor/github.com/kurin/blazer/b2/b2.go:563 +0x72
github.com/kurin/blazer/b2.(*Object).Attrs(0xc0005e29a0, 0xf20fe0, 0xc00013d8c0, 0x0, 0x0, 0xdca1b6)
	/restic/vendor/github.com/kurin/blazer/b2/b2.go:443 +0x5d
github.com/restic/restic/internal/backend/b2.(*b2Backend).Stat(0xc00021c500, 0xf20fe0, 0xc00013d8c0, 0xdca1b6, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/restic/internal/backend/b2/b2.go:223 +0x179
main.open(0xc00003a102, 0x13, 0xc00003a102, 0x13, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/restic/cmd/restic/global.go:650 +0x533
main.OpenRepository(0xc00003a102, 0x13, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x2, ...)
	/restic/cmd/restic/global.go:358 +0x96
main.runBackup(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000338ee0, ...)
	/restic/cmd/restic/cmd_backup.go:413 +0x2da
main.glob..func2(0x14ab4e0, 0xc00013dac0, 0x0, 0x4, 0x0, 0x0)
	/restic/cmd/restic/cmd_backup.go:63 +0x1bf
github.com/spf13/cobra.(*Command).execute(0x14ab4e0, 0xc00013da40, 0x4, 0x4, 0x14ab4e0, 0xc00013da40)
	/restic/vendor/github.com/spf13/cobra/command.go:762 +0x460
github.com/spf13/cobra.(*Command).ExecuteC(0x14adfa0, 0xc00031feb0, 0x42d511, 0xe0fd98)
	/restic/vendor/github.com/spf13/cobra/command.go:852 +0x2ea
github.com/spf13/cobra.(*Command).Execute(...)
	/restic/vendor/github.com/spf13/cobra/command.go:800
main.main()
	/restic/cmd/restic/main.go:86 +0x3f

goroutine 6 [syscall]:
os/signal.signal_recv(0x0)
	/usr/local/go/src/runtime/sigqueue.go:147 +0x9c
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:29 +0x41

goroutine 7 [chan receive]:
github.com/restic/restic/internal/restic.init.0.func1.1()
	/restic/internal/restic/lock.go:257 +0xb3
created by github.com/restic/restic/internal/restic.init.0.func1
	/restic/internal/restic/lock.go:254 +0x35

goroutine 17 [chan receive]:
github.com/restic/restic/internal/restic.init.2.func1(0xc000096120)
	/restic/internal/restic/progress_unix.go:17 +0x63
created by github.com/restic/restic/internal/restic.init.2
	/restic/internal/restic/progress_unix.go:16 +0xab

goroutine 20 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc000140eb0)
	/restic/vendor/go.opencensus.io/stats/view/worker.go:154 +0x100
created by go.opencensus.io/stats/view.init.0
	/restic/vendor/go.opencensus.io/stats/view/worker.go:32 +0x57

goroutine 21 [chan receive]:
main.CleanupHandler(0xc0001c8a20)
	/restic/cmd/restic/cleanup.go:62 +0x5d
created by main.init.0
	/restic/cmd/restic/cleanup.go:24 +0x67

goroutine 22 [select]:
github.com/restic/restic/internal/ui/termstatus.(*Terminal).run(0xc0001376e0, 0xf20fe0, 0xc000222140)
	/restic/internal/ui/termstatus/status.go:98 +0x16e
github.com/restic/restic/internal/ui/termstatus.(*Terminal).Run(0xc0001376e0, 0xf20fe0, 0xc000222140)
	/restic/internal/ui/termstatus/status.go:83 +0x7d
main.glob..func2.1(0x0, 0x0)
	/restic/cmd/restic/cmd_backup.go:61 +0x6d
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000140f50, 0xc000358320)
	/restic/vendor/gopkg.in/tomb.v2/tomb.go:163 +0x2b
created by gopkg.in/tomb%2ev2.(*Tomb).Go
	/restic/vendor/gopkg.in/tomb.v2/tomb.go:159 +0xc7

goroutine 191 [IO wait]:
internal/poll.runtime_pollWait(0x7f92b6b5df88, 0x77, 0xc0000eb200)
	/usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc0003d2598, 0x77, 0xf21000, 0xc0003bb020, 0xc0003d2580)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitWrite(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).WaitWrite(...)
	/usr/local/go/src/internal/poll/fd_unix.go:498
net.(*netFD).connect(0xc0003d2580, 0xf21060, 0xc0003bb020, 0x0, 0x0, 0xf0d9a0, 0xc0004ccd00, 0x0, 0x0, 0x0, ...)
	/usr/local/go/src/net/fd_unix.go:152 +0x29d
net.(*netFD).dial(0xc0003d2580, 0xf21060, 0xc0003bb020, 0xf26d20, 0x0, 0xf26d20, 0xc0003d67e0, 0x0, 0x0, 0x40c638)
	/usr/local/go/src/net/sock_posix.go:149 +0xff
net.socket(0xf21060, 0xc0003bb020, 0xdc8154, 0x3, 0x2, 0x1, 0x0, 0x0, 0xf26d20, 0x0, ...)
	/usr/local/go/src/net/sock_posix.go:70 +0x1c0
net.internetSocket(0xf21060, 0xc0003bb020, 0xdc8154, 0x3, 0xf26d20, 0x0, 0xf26d20, 0xc0003d67e0, 0x1, 0x0, ...)
	/usr/local/go/src/net/ipsock_posix.go:141 +0x141
net.(*sysDialer).doDialTCP(0xc0003d2500, 0xf21060, 0xc0003bb020, 0x0, 0xc0003d67e0, 0xca3c80, 0x14d6498, 0x0)
	/usr/local/go/src/net/tcpsock_posix.go:65 +0xc2
net.(*sysDialer).dialTCP(0xc0003d2500, 0xf21060, 0xc0003bb020, 0x0, 0xc0003d67e0, 0xc0005c9460, 0x10, 0x10)
	/usr/local/go/src/net/tcpsock_posix.go:61 +0xd7
net.(*sysDialer).dialSingle(0xc0003d2500, 0xf21060, 0xc0003bb020, 0xf162a0, 0xc0003d67e0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/dial.go:578 +0x36e
net.(*sysDialer).dialSerial(0xc0003d2500, 0xf21060, 0xc0003bb020, 0xc0005c9460, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/dial.go:546 +0x221
net.(*Dialer).DialContext(0xc0000f6480, 0xf20fe0, 0xc00013d8c0, 0xdc8154, 0x3, 0xc0004cc7c0, 0x18, 0x0, 0x0, 0x0, ...)
	/usr/local/go/src/net/dial.go:424 +0x666
net/http.(*Transport).dial(0xc000388000, 0xf20fe0, 0xc00013d8c0, 0xdc8154, 0x3, 0xc0004cc7c0, 0x18, 0x0, 0x640d30, 0xc0002cf600, ...)
	/usr/local/go/src/net/http/transport.go:1040 +0x1f5
net/http.(*Transport).dialConn(0xc000388000, 0xf20fe0, 0xc00013d8c0, 0x0, 0xc0004f4e80, 0x5, 0xc0004cc7c0, 0x18, 0x0, 0xe0fd98, ...)
	/usr/local/go/src/net/http/transport.go:1478 +0x193e
net/http.(*Transport).dialConnFor(0xc000388000, 0xc0001e8f20)
	/usr/local/go/src/net/http/transport.go:1308 +0xdc
created by net/http.(*Transport).queueForDial
	/usr/local/go/src/net/http/transport.go:1277 +0x41d

goroutine 227 [select]:
net.(*netFD).connect.func2(0xf21060, 0xc0003bb020, 0xc0003d2580, 0xc00035cde0, 0xc00035cd80)
	/usr/local/go/src/net/fd_unix.go:129 +0xba
created by net.(*netFD).connect
	/usr/local/go/src/net/fd_unix.go:128 +0x275

rax    0xfffffffffffffffc
rbx    0x7012b86
rcx    0x45d1f3
rdx    0x0
rdi    0x14bdbc0
rsi    0x80
rbp    0xc0000536e8
rsp    0xc0000536a0
r8     0x0
r9     0x0
r10    0xc0000536d8
r11    0x202
r12    0x762a45f445083
r13    0x1
r14    0x762a45f445083
r15    0x0
rip    0x45d1f3
rflags 0x202
cs     0x33
fs     0x0
gs     0x0