I’ve been using restic successfully for a while. I have been using the Backblaze B2 API/backend and have a lot of data in my current repo in a B2 bucket. I saw this warning and I wanted to ask for a little more detail to decide if I need to switch to using the S3 API.
The docs say “Due to issues with error handling in the current B2 library that restic uses, the recommended way to utilize Backblaze B2 is by using its S3-compatible API.”
Is this a short term problem and is there an updated B2 library now that provides the error handling? If so, is there a plan to update restic with the new library so B2 can be fully supported?
What is the risk of continuing to use the B2 API? Data corruption? If so, does "check’ catch it or would I need to do “check --read-data”?
If I need to move from B2 to S3, is there a way to convert my existing repo or would I need to back everything up again if I change to S3?
The way forward with B2 is to use S3. It’s well supported by B2 so there’s really no reason not to. Just go for it. The contents of the repository is still the same so you should be able to just change the repository URI unless I’m terribly mistaken (I don’t use either of them myself). No need to convert anything.
I’ve been using Backblaze as a storage backend for years now and when I started using it there was definitely no warnings. As asked by @DigitalMan already, is there somewhere some resources as what could go wrong when using Backblaze backend without S3-compatibility layer?
I’ve tried to start using S3-API on Backblaze, but can’t do that.
Previously I had restic repo specified like this: b2:BUCKET_NAME:DIRECTORY
Reading various documentation, I’ve tried to use this instead (in addition to changing environment variable names from Backblaze to AWS specific): s3:s3.eu-central-003.backblazeb2.com/BUCKET_NAME/DIRECTORY
Using the change above, restic spits out an error:
Fatal: unable to open config file: Stat: Access Denied.
Is there a repository at the following location?
Now, I was unable to find region eu-central-003 from anywhere except from some old Reddit post - is there a better way to find a region of my bucket region URL in Backblaze? I definitely have buckets at EU region, but maybe it’s not correctly specified.
Any other ideas how to make restic work with S3-API on Backblaze to not be affected by the aforementioned warning on restic documentation?
Thanks! For me it’s -- and I can click on the info icon to see this text:
Buckets created before May 4, 2020, cannot be used with our S3 Compatible APIs.
So it seems that I have to create a new bucket to use S3-compatible API. Not sure if it’s possible to migrate data without extra cost from one bucket to another at Backblaze. Any ideas?
I, also not aware of this warning by now, use restic with b2 for years and never had any problems. I also did restores already without issues. I am sure other people here can confirm the same.
For some reason still last year I tried to use backblaze over s3. But I didn’t like this thing that unused data then gets not deleted but only “hidden”. I found that confusing and didn’t want to put any bucket setting to delete hidden files. So I stayed with b2.
As long as your current B2 configuration works, there’s no need to switch to S3. But if something doesn’t work then the S3 backend will provide error messages instead of just silently hanging.
When you log into the b2 web ui you click on lifecycle rules of the bucket your using. Then choose Keep only the last version of the file. Then click update bucket. You’re done. Same behavior as before.
This is a default safety feature of object storage of many vendors.
Thanks for sharing the article. Reading that I’m a little confused - you’re writing there that:
It will do the server-side copying where possible.
What does it mean in practice when it comes to additional cost from Backblaze for the migration? Can you remember what size of bucket did you migrate and how extra cost did it add to your Backblaze bill in the end?
As long as you use rclone 1.49 or later and you copy between two buckets that belong to the same account you should only have to pay for the transactions. They’re pretty cheap.
It’s probably falling mostly under the most expensive Transactions Class C right? I have multiple buckets for restic totalling ~7TB and would like to understand how much this migration is going to cost for me approximately before taking this road.
So as I follow this conversation, it is transaction class C, and each file is one API call. So this would be approximately in total: cost(number of files) = 0.004$ * (number of files-2500)/1000 (+ some other calls, which I cannot estimate and assuming you have more than 2500 files)
Great question. The cost of the storage while there are temporarily two copies of the data will likely be the higher cost.
In my case I had a 1.5 TB repository, 276,015 files with mostly 4 MB pack size. The equation MelcomX gave yields $1.09. For the month that I did the transfer and a bunch of testing, my bill had $1.32 in class C transaction costs.
This b2 go library was taken over by Backblaze themselves and then have not maintained it and suggest to use the s3 protocol.
Blazer targets the Backblaze B2 Native API. Unless you specifically need to access Backblaze B2 via its Native API, you should use the MinIO Go Client SDKwith Backblaze B2’s S3 Compatible SDK
After all the years of saying that their protocol was much cheaper to implement without load balancers…
i also use b2 protocol at the moment. backups are working, i can do restic mount.
i can change the B2-Vars to S3 and switching to s3 backend seems simple. i have tested it with list snapshots and mounts.
i am a bit uncertain if i should change from b2 to s3