Restic stopped connecting to backblaze

Getting Help
1

Hello,
I’ve been using restic with backblaze for about a year now and for the most part it worked fine. Yesterday all of a sudden it stopped connecting to backblaze. It just hangs. No error produced.

root@backup1:~/bin/backup# ./restic -r b2::/backup/ns1 --verbose=3 snapshots
signal interrupt received, cleaning up

It just stays like this forever until I do Ctrl-C. All credentials variables are set properly - again it worked for almost a year and suddenly stopped yesterday.
Sniffing traffic with tcpdump I see some continuous exchange with api.backblazeb2.com, but it doesn’t go anywhere.

I checked my backblaze account works fine - I downloaded their b2 command-line utility and am able to access my bucket with the same credentials restic is using.
I’m not sure how to troubleshoot it any further.
Is anybody aware of any changes with backblaze? Do you have any hints?
Thanks,
Michael

2

For what it’s worth: My backups continue to work flawlessly with backblaze.

3

Looks like your repository URL is formatted incorrectly - it’s missing the bucket name. Unless I’m mistaken, please fix that and try again to see if the problem persists.

Which version of restic are you currently using? Please update to the latest release (version 0.12.1) and try again to see if the problem persists.

If the problem does persist, please build restic with debug enabled and enable debug logging. Then look into that file to see if you can spot where restic stops.

4

Good to know. Thanks

5

Repository URL is correct. When I wrote the message i specified it as “<” bucket-name “>” and for some reason this forum has eaten the content between angle brackets. Anyway URL is b2:bucketname/backup/ns1 and like I said it worked for a long time.
I’m using latest 0.12.1 version. Thanks for the hint for enabling debugging. This is what I’ll do next. Then I’ll report back.
Thanks

6

Problem solved. I needed to update list of trusted root certificates. Apparently backblaze changed ssl certificate 2 days ago and restic was unable to establish https connection. Looking in the log it was very apparent saying something like “certificate signed with unknown authority”.
Thank you rawtaz for the hint on enabling debugging. I didn’t know how to do it.
Michael

1 Like
7

Hi,

My backup stopped working too 2 days ago from my CENTOS server to backblaze. When I do netstat when trying to restic snapshots … all I see is TIME_WAIT. Looks like the server is unable to establish connection to backblaze. How did you exactly update the list of trusted root certificate?

Thanks.
Ross

8

I’m not sure how to do it exactly on centos. There should be a package that provides root certificates, but I don’t know the package name for centos. Certificates are just text files so it should be safe just to take it from the latest centos version and install it. That’s what I did on my backup machine running old slackware.

9

Thanks for your reply. I looks like this got something more to do with letsencrypt w/c B2 is using. Centos 6 openssl version 1.0 is having issue with it. Maybe restic is using openssl to verify certificates and it just keeps hanging. I can’t do any restic command at all… it just hangs. But when I use restic from my Macbook… it’s fine… I can connect to my B2 repository.

[EDIT] I figured a workaround. My Centos 6 box was unable to authenticate the certificate from backblaze with letsencrypt. I just used --cacert option with restic and I was able to connect to backblaze. Thanks.

10

Old root certificates from Let’s Encrypt expired on September 30th. Sounds like your system is still using those old root certificates by default.

11

According to About/Product - CentOS Wiki CentOS 6 is not getting updates since nearly one year. So no wonder your system is missing the updated certificates.