Very Large Number of B2 Transactions

Hey,

I’m sorry that you had an unexpected B2 bill. Did you know that you can configure B2 to alert you once a specified number of (non-free) transactions have been executed?

Back to restic: In the 0.8.3 release we’ve merged several improvements for cloud-based backends and B2 especially. With the help of the library author we’re using to access B2, we were able to reduce using the API call b2_download_file_by_name a lot. During regular operation with restic < 0.8.3, this API call was also used during operations such as backup. Once we learned that we were able to work around it. A large portion of the costly API requests were likely caused by this.

Since 0.8.0, restic has a local metadata cache, which speeds up operations and reduces backend API requests for metadata. If you have created the repository with restic < 0.8.0, it may happen that there’s data in the repo which isn’t cached, and requested over and over. You can correct this by using restic prune. If you’ve already done that in the past, all should be fine.

The next thing that you may not be aware of: By default, restic check does not use a cache. In general, I think this is a good idea: It’ll notify you of files changed on the server, even if the unmodified file is still in your cache. The downside is that all data is fetched on demand from the repo, which may also cause many transactions because restic’s check process is rather dumb at the moment: Every tiny blob of data is requested separately, so if five blobs are needed from a single files in the repo, restic will send five separate requests.

You could pass --with-cache to restic check, but probably that’s not a good idea because it’ll take the cache for granted and not re-download files. I’m thinking about using a temporary directory as the cache for restic check, but that hasn’t been implemented yet.

3 Likes