I’ve been using restic for ~1 year, backing up to several repos on B2. Recently the restic command just hangs and does not output any verbose info when connecting to the B2 repos. My last successful daily backup was on Sep. 30; hanging issue started on Oct. 1.
I have verified the following:
My B2 account is in good standing
I can log into Backblaze web interface and view all files
I can use the B2-supplied command line tool to download files from the same client machine
Restic returns some basic info from restic --help or restic cat
I have deleted the lock directory from all repos. After none of this worked, I have also deleted my local cache.
Interestingly the restic check command started hanging a couple days earlier. It ran fine on Sep 29, then started hanging later that day.
Has anyone else run into this issue? Did something change very recently in the B2 interface that restic uses?
I might have a related issue with a B2 repo (via rclone). I’m not sure whether its related to rclone v1.56.2 (installed around the same time) or something changed at B2.
About once a day a cron running rectic check (restic v0.12.1) gets blocked by already locked exclusively... errors. The cron was working smoothly until 2021-10-01. Unlocking the repo manually will allow it to run a few more times before the below lock-deletion error occurs again:
rclone: 2021/10/06 00:46:42 ERROR : locks/XXXXXXXX: Delete request remove error: failed to delete "locks/XXXXXXXX": File not present: locks/XXXXXXXX (400 file_not_present)
Remove(<lock/XXXXXXXX>) returned error, retrying after 13.516432903s: blob not removed, server response: 500 Internal Server Error (500)
error in cleanup handler: blob not removed, server response: 500 Internal Server Error (500)
Adding to the mystery, restic forget --prune will sometimes (since 2021-10-01) fail with the below error, but re-running the command a few minutes later will sometimes show the snapshot files were in-fact deleted. B2 might have an intermittent coherency issue since 2021-10-01…?
rclone: 2021/10/09 03:25:27 ERROR : snapshots/XXXXXXXX: Delete request remove error: failed to delete "snapshots/XXXXXXXX": File not present: snapshots/XXXXXXXX (400 file_not_present)
Remove(<snapshot/XXXXXXXX>) returned error, retrying after 873.42004ms: blob not removed, server response: 500 Internal Server Error (500)
unable to remove <snapshot/XXXXXXXX> from the repository
[1:02] 79.17% 19 / 24 files deleted
blob not removed, server response: 500 Internal Server Error (500)
github.com/restic/restic/internal/backend/rest.(*Backend).Remove
/restic/internal/backend/rest/rest.go:381
github.com/restic/restic/internal/backend.(*RetryBackend).Remove.func1
/restic/internal/backend/backend_retry.go:108
github.com/cenkalti/backoff/v4.RetryNotifyWithTimer
/home/build/go/pkg/mod/github.com/cenkalti/backoff/v4@v4.1.1/retry.go:55
github.com/cenkalti/backoff/v4.RetryNotify
/home/build/go/pkg/mod/github.com/cenkalti/backoff/v4@v4.1.1/retry.go:34
github.com/restic/restic/internal/backend.(*RetryBackend).retry
/restic/internal/backend/backend_retry.go:46
github.com/restic/restic/internal/backend.(*RetryBackend).Remove
/restic/internal/backend/backend_retry.go:107
github.com/restic/restic/internal/cache.(*Backend).Remove
/restic/internal/cache/backend.go:38
main.deleteFiles.func2
/restic/cmd/restic/delete.go:47
golang.org/x/sync/errgroup.(*Group).Go.func1
/home/build/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1371
I can’t help but wonder if the recent B2 issue reports have anything to do with the SSL certificate changes on the Internet that took place last day of september.
Yes, this makes a lot of sense. Updating the trusted SSL certificates list on my local machine is making all the difference. Backups to B2 are working again!
This help forum is a great companion to restic. Thanks, all, for the quick responses…and particularly @MichaelEischer & @rawtaz for pointing me to the recent changes to handling SSL certificates.
Coincidentally, this appears to be the cause of the other issue mentioned in my opening post. One of my backup scripts is now frequently running into the 400/404 error when trying to delete a lock that does not exist (presumably already deleted on the first try, but not properly reported by B2 as a successful operation). This started on 2021-09-29.