Restic hangs with B2 since 2021-10-01

I’ve been using restic for ~1 year, backing up to several repos on B2. Recently the restic command just hangs and does not output any verbose info when connecting to the B2 repos. My last successful daily backup was on Sep. 30; hanging issue started on Oct. 1.

I have verified the following:

  • My B2 account is in good standing
  • I can log into Backblaze web interface and view all files
  • I can use the B2-supplied command line tool to download files from the same client machine
  • Restic returns some basic info from restic --help or restic cat

I have deleted the lock directory from all repos. After none of this worked, I have also deleted my local cache.

Interestingly the restic check command started hanging a couple days earlier. It ran fine on Sep 29, then started hanging later that day.

Has anyone else run into this issue? Did something change very recently in the B2 interface that restic uses?

I might have a related issue with a B2 repo (via rclone). I’m not sure whether its related to rclone v1.56.2 (installed around the same time) or something changed at B2.

About once a day a cron running rectic check (restic v0.12.1) gets blocked by already locked exclusively... errors. The cron was working smoothly until 2021-10-01. Unlocking the repo manually will allow it to run a few more times before the below lock-deletion error occurs again:

rclone: 2021/10/06 00:46:42 ERROR : locks/XXXXXXXX: Delete request remove error: failed to delete "locks/XXXXXXXX": File not present: locks/XXXXXXXX (400 file_not_present)
Remove(<lock/XXXXXXXX>) returned error, retrying after 13.516432903s: blob not removed, server response: 500 Internal Server Error (500)
error in cleanup handler: blob not removed, server response: 500 Internal Server Error (500)

Adding to the mystery, restic forget --prune will sometimes (since 2021-10-01) fail with the below error, but re-running the command a few minutes later will sometimes show the snapshot files were in-fact deleted. B2 might have an intermittent coherency issue since 2021-10-01…?

rclone: 2021/10/09 03:25:27 ERROR : snapshots/XXXXXXXX: Delete request remove error: failed to delete "snapshots/XXXXXXXX": File not present: snapshots/XXXXXXXX (400 file_not_present)
Remove(<snapshot/XXXXXXXX>) returned error, retrying after 873.42004ms: blob not removed, server response: 500 Internal Server Error (500)
unable to remove <snapshot/XXXXXXXX> from the repository
[1:02] 79.17%  19 / 24 files deleted
blob not removed, server response: 500 Internal Server Error (500)*Backend).Remove

I can’t help but wonder if the recent B2 issue reports have anything to do with the SSL certificate changes on the Internet that took place last day of september.

1 Like

@padraicshafer Your issue sounds a lot like Restic stopped connecting to backblaze - #10 by 764287

@stephenlb That looks similar to the B2 issue in .


Yes, this makes a lot of sense. Updating the trusted SSL certificates list on my local machine is making all the difference. Backups to B2 are working again!

This is indeed the same issue and solution as in Restic stopped connecting to backblaze.

For those with Synology DSM, here are the commands I used, based on b2: Failed to initialize gateway with error x509: certificate signed by unknown authority and Import root certificate in Synology.

openssl s_client -showcerts -connect > /usr/share/ca-certificates/b2.crt
ln -s /usr/share/ca-certificates/b2.crt /etc/ssl/certs/b2.crt
1 Like

This help forum is a great companion to restic. Thanks, all, for the quick responses…and particularly @MichaelEischer & @rawtaz for pointing me to the recent changes to handling SSL certificates.

Had very similar problems. Didn’t touch anything about SSL root certificates but started working again about 48hrs ago

Coincidentally, this appears to be the cause of the other issue mentioned in my opening post. One of my backup scripts is now frequently running into the 400/404 error when trying to delete a lock that does not exist (presumably already deleted on the first try, but not properly reported by B2 as a successful operation). This started on 2021-09-29.

I’m looking forward to trying out the update in PR #3544 (b2: Successful delete if file does not exist).