Maybe I just figured it wrong, but after running backups on a regular schedule my B2 costs are quite a bit higher than I expected. Perhaps someone can clarify this for me.
Some stats straight from the Backblaze panel:
Bucket size: 5797.3 GB
Current Files: 1,127,178
File Lifecycle: Keep only the last version
Encryption: Disabled
I have about 5.5 TB of actual data so the bucket size seems about right. My problem lies with the Download Bandwidth they charge you for. My current usage for today is $0.24 (25 GB), according to their billing panel (I think it’s just 1 cent/GB). Today I only ran one backup, for mysqldump'ing a bunch of databases. All my backups are tagged, so I can just run restic stats --mode raw-data --tag backup_mysql to get the total size used by just these dumps. My retention is set to --keep-daily 60 and this comes out to 13.252 GiB, so that’s maybe a few 100 MB churn per day.
Now here comes the fun: for the past month they charged me $13.59 for bandwidth, at 1 cent per GB that means restic downloaded well over a terabyte just for running backups. Now 15 bucks isn’t a lot, but as the repo grows so will this bandwidth usage and eventually I will probably be paying multiples of that just for running backups (besides the actual storage cost). Somehow that just doesn’t seem right to me. :>
I run backup, forget, prune, check and cache --cleanup every time. I think you’re implying that the cache constantly gets rebuilt and it has to download data from the repo? So I checked the size of the cache dir (du -hs) and it’s about 23 GB, roughly the same as my bandwidth usage. I thought caching was used for like keeping track of the current backup’s state, and not so much across multiple backups. But apparently that’s not the case?
check by default uses a fresh temporary cache and downloads all directory metadata in a repository, which have approximately the size of the cache directory. So that would be 23GB per check run.
Then depending on the restic version, prune can also consume large amounts of bandwidth. Before restic 0.12.0 prune removes every single unnecessary byte from the repository which means that it has to download gigabytes of data from the repository and then also upload it again. Since restic 0.12.0 this has been optimized to allow up to 5% “garbage” data within the repository, and thus fewer data files need to be rewritten. There are also parameters allowing you to increase that fraction. This will in turn reduce the amount of repository data restic has to rewrite.
I’d also recommend to run prune only e.g. once per week and not after every backup. The same basically also applies for check.
I see, I figured you’d want to run prune every time to keep the repo size as small as possible. As for check, is it actually necessary? Like, if backups complete successfully (not even a warning) I think you can basically assume the repo’s integrity is in order? So maybe running it once a month would be sufficient? I haven’t seen a single error from check so far anyways.
And it’s “alright” to run forget after every backup isn’t it? Because the snapshots’ information should already be present in the cache and all it does is mark some of them for deletion. So it shouldn’t be pulling any data from the repo, or at least not anything worth mentioning.
You could aim for that, but if you feel that you need to keep the disk space usage down that much, you probably have bigger issues
There’s plenty of things that can go wrong, there’s been instances where B2 corrupted data on their end and there’s the possibility of bit rot. So running check every now and then makes sense, but you cannot assume that just because a backup run succeeded, the entire repo is still intact since your last check. Backing up only adds data to the repository, it doesn’t check any of the other data. So no, a successful backup run does not tell you that there hasn’t been bit rot or similar on the storage side.
That’s not uncommon or bad. There’s no specific answer to how often to run it - it depends on how susceptible your storage is to corruption. If you have a solid and redundant storage then running check rarely is fine, if you have a problematic storage you might want to check the repository more often. Also note that you sometimes might want to check with --read-data to make sure that all of the actual data is intact as well (not just the integrity of the repository itself).
You can. All of this, including both check and prune is entirely up to you. You can run it every five minutes if you want. It all boils down to how often it makes sense to do it. Personally I don’t see a reason to run prune after every backup run. How much disk space do you actually gain by that, and how much does it cost you in terms of processing traffic with B2? If you don’t have a reason do to it that often, I don’t see the point of doing so. Prune will rewrite some stuff like indexes and pack files.
Good point, although luckily that hasn’t happened to me so far. :>
Seems reasonable. Is there any way I can check how much data it would read? Or is the amount basically the output from restic stats --mode raw-data?
And just to confirm: last night 3 backups were made and my current download usage according to Backblaze is now $0.00 (16 KB) (this is with just backup and forget). The night before I only had 1, for which I still let restic do check and the usage was still/again around 23 GB. So it looks like my problem was indeed having check run after every backup. Thanks @MichaelEischer. Guess I’ll just start with running these maintenance tasks every 2 weeks and see how it goes. =]
Personally I wouldn’t bother with the forget until you are ready to prune. You don’t save anything in terms of $, and you never know when that one time occurs when you may need something that is technically still backed up but is not in any of your retained snapshots.
check --read-data will read the whole repository. Its probably easier to just look at how much data is storead at B2. To only read a fraction you could use check --read-data-subset x/y.
Once data was correctly uploaded to B2 it will very likely stay that way (although we’ve seen some temporary problems from time to time, which are hopefully fixed by now). So check would mostly guard against bugs in restic and bitflips on the computer running the backup.
