Hi! New restic user here. I played around with restic and getting backups working to a cloud provider. After doing some successful small backups, I was ready to start a full backup consisting of around 180 GB.
The backup was running fine. It took over 24 hours with my current Internet connection, but I realized with an estimated 2 hours remaining in the backup that I forgot about a directory that I wanted backed up. After doing some reading, I thought that I could interrupt the backup and it could continue where it left off. But it sounds like that might only happen if there is an initial snapshot of the data already?
In any case, I interrupted the backup (control-C), added the missing directory, and restarted it. However, it appeared that the backup was starting from scratch. Ugh. That was a waste of the last 24 hours and its associated bandwidth. In any case, I started another backup, this time consisting of about 130 GB of the original backup. That backup completed fine, but when I go to check my repository it now says that the repository size is over 400 GB in size?!
I ran a “restic forget --keep-only 1” and a “restic prune” thinking that it would just keep this one copy of my latest 130 GB backup and delete everything else, but to no avail. Somehow my repo size is 3x of what I would expect it to be?
At my repository (backblaze) settings, I ensured that the lifecycle is set to only keep the latest copy of data so I don’t think that multiple copies are being held there. I can’t think of why else my repo would be 3x larger than my data except that it’s holding onto that initial interrupted backup data and can’t clean that up? Maybe since the backup was interrupted, it doesn’t have any record of it and can’t safely delete it?
Do I have any hope of deleting this extra space and reclaiming it? I don’t want to have to pay for data in my repo just sitting there not referenced by any backup and just being wasted space. Besides deleting my existing repo and starting over, is there any hope of cleaning up this wasted space? I’m a bit hesitant to continue backing more things up until I can get a handle on my space usage in my current repo. I’m trying to keep costs down and not have to pay for space that isn’t really being used.
I don’t have any experience with Backblaze but I have a few questions:
Are you sure you don’t have any references to other folders in your backup data? If you’re not sure, you can use the --one-file-system parameter during backup to make sure you don’t accidentally backup files from other file systems.
Can you mount the backup and peek inside what’s there?
I’m sure I don’t have any nested filesystems under the path that I’m backing up.
I am able to mount the backup and I see the just one snapshot (when browsing the “snapshots/” directory). I don’t see any other snapshots or anything that would represent the wasted data. Although I’m guessing if restic knew about it, it would be able to take care of it via prune.
Backblaze is who is telling me that the size is 400GB. When issuing “restic stats --mode raw-data”, it shows me that total size is “134.038 GiB”. But looking at the backblaze website, the size of the storage bucket is “403.6 GB”. Perhaps backblaze isn’t actually deleting files despite restic telling it to prune files? I’m not sure if that is the case, but it’s certainly a possibility.
“restic snapshots” shows me just the one snapshot that I would expect to see. I have deleted all other snapshots until I could figure out how to properly reclaim space.
At this point, it’s either that restic has lost track of some uploaded data and hence isn’t able to prune it because it doesn’t know that it’s there? Or that backblaze isn’t actually deleting files despite restic telling it to because of how it handles things on their side.
At this point, I think I’m going to wipe out my backblaze data and just start over. I’ll do more experimenting with them since they give me 10GB of free data to start with. I’ll do some smaller backups and see if I can replicate the growing storage problem there, but under the 10 GB limit so I don’t incur fees while doing so.
Ok that sounds a) like you know what you’re doing and b) like the problem is on Backblaze’s side. Maybe someone here with more Backblaze experience can help but other than that your strategy sounds fair to me.
Interrupting a backup and starting over is very robust. And while it started over, the second backup would have only uploaded files it hadn’t yet uploaded. But it has to start reading your all your files to make sure they are all stored, which is when the snapshot is finally recorded. It’s worthwhile to take a little time to read the design document that steps through how restic works. It really helps to understand what is happening and makes it much less like a black box. This will give you more confidence in what is happening, especially for backing up your files.
Finally, do a search on the forums for posts relating to backblaze. You should find plenty of shared tips and advice using that backend with restic.
I concur with you both. Doing a little more testing, it appears to be some kind of behavior with backblaze.
I just started with a clean repository and uploaded 1 GB to it via various restic backup snapshots. After forgetting and pruning about 1/2 of them, backblaze is still showing that I’m using 1 GB. I done more reading but haven’t found any posts that describe exactly what is happening here. Just a few posts that say that backblaze seems to resolve the space discrepancy after a few days (although I did wait almost 48 hours and never saw any improvement). In any case, I’m going to just let the repo sit as is and see if it does resolve itself and how long it takes.
In the mean time, I did another test using an sftp repo target and that repo behaved much more like I expected it to, although not entirely perfect? Creating a repo there and testing some backups and forget/prune commands, I see that things are getting backed up to the repo and deleted upon being pruned.
Still, when I interrupt a backup and then go to prune, it appears that there are still things in repo:data/ that aren’t entirely cleaned up and get left behind. For example, I backed up a directory that was 288 MB. Upon completion, I did a forget and prune of that snapshot and I got all of my 288 MB back.
However, I did a backup of that same directory but this time interrupted it mid way through so that it only uploaded 163 MB. After doing a prune, it deleted 100 MB of this data but left the other 62 MB in the repo. I still need to play around with it some more to see if it is possible for restic to locate and clean up the rest of this “lost” data. Or perhaps such is life when dealing with interrupted backups, which shouldn’t be frequent in real world cases too often but probably aren’t entirely avoidable (i.e. network interruptions, power outages, client/server crashes, etc). It would still be nice to identify and get everything perfectly clean so that no space is wasted. 62 MB of space wasted in this case isn’t that much in the greater scheme of things. But I don’t know if that lost space increases proportionally with the size of the backups themselves. I’ll have to do more testing on that front.
I once had something like this happen to me on Backblaze. I deleted an entire bucket and it took a couple of days to reflect the space savings but I didn’t have to do anything on my part.
I once had something like this happen to me on Backblaze. I deleted an entire bucket and it took a couple of days to reflect the space savings but I didn’t have to do anything on my part.
Good to know. When you say your deleted an entire bucket, I’m assuming you emptied out your repository via restic? And not actually deleting the bucket via the backblaze interface?
Like I mentioned above, I’m going to let that backblaze repo sit for a few days and see if it ultimately doesn’t end up showing the correct repo size after a few days to reflect the data that was forgotten/pruned.
Ok so then it seems we have found the problem! Good to know. Regarding the space recovery during pruning, please read this part of the manual. Spoiler: pruning is not 100% efficient by default (efficiency vs. speed and such).
That said, interrupting a backup run does not create any problems. restic just goes through all files again and only uploads the chunks it hasn’t already uploaded. Only when all chunks are successfully uploaded, they are then referenced as a snapshot. restic doesn’t care how many attempts it needs to get there.
That said, interrupting a backup run does not create any problems. restic just goes through all files again and only uploads the chunks it hasn’t already uploaded. Only when all chunks are successfully uploaded, they are then referenced as a snapshot. restic doesn’t care how many attempts it needs to get there.
If I understand things correctly, restic does need to index those chunks that it has uploaded first before it can ignore them for a subsequent backup? And maybe this interim indexing only happens every so often during the backup?
I’ve been testing a backup of ~300MB of data which takes 2 minutes and 28 seconds on my current connection to complete. I then did a forget/purge on that snapshot and tried things again. This time, I let the backup get around 80+% complete before interrupting it (control-C). I then restarted the backup to see how long it would take to finish and took exactly 2 minutes and 28 seconds again. I would have thought that with 80+% complete, that it would only have to upload the remaining 20%. But perhaps that data never hit an index and/or there was no snapshot record of it, so restic didn’t really know about those pieces to be able to continue?
Back to my issue of “lost” files in the data directory, it appears that all of those files that are in there that can’t be removed via prune have the naming convention:
[hash1]-restic-temp-[hash2]
While items that can be pruned do not have the trailing “-restic-temp-[hash2]”. A few examples from my repo:
I’m assuming that files that include -restic-temp- are files that were partially uploaded but never written as a complete pack? If I remove all my snapshots and do a prune, the only data that is left behind in data/ seems to be all -restic-temp- files that take up space but that restic seemingly does not clean up.
Is there any harm in manually deleting these seemingly stale files? They only seem to take up maybe ~60 MB or so for each interrupted backup. But the OCD part of me wants to have my repo not wasting any space.
Due to issues with error handling in the current B2 library that restic uses, the recommended way to utilize Backblaze B2 is by using its S3-compatible API.
Follow the documentation to generate S3-compatible access keys and then setup restic as described at Amazon S3. This is expected to work better than using the Backblaze B2 backend directly.
I did set up the repo using the S3-compatible API like the docs recommended.
I also made note of:
Different from the B2 backend, restic’s S3 backend will only hide no longer necessary files. By default, Backblaze B2 retains all of the different versions of the files and “hides” the older versions. Thus, to free space occupied by hidden files, it is recommended to use the B2 lifecycle “Keep only the last version of the file”. The previous version of the file is “hidden” for one day and then deleted automatically by B2.
My repo already had “Keep only the last version of the file” set. I did way over a day (probably closer to 48 hours) to see if those files would have been automatically deleted, but they were not. I’m going to let them sit for a whole week and try to remember to check daily to see if they do end up clearing up eventually.
There will nearly always be extraneous data stored in your repository, unless you prune with --max-unused 0
That operation could generate considerable API requests to your cloud provider, and can take some time. The default max unused storage is 5%.
Have a read of the Restic Design Document. You don’t necessarily have to understand of the bibs and bobs, but the broad principles of how restic works really helps to understand its behaviour better. And give you more confidence in your backups.
I did prune initially with --max-unused 0 to see if that was causing the left over data that I was seeing, but it made no difference in my particular situation.
At this point, my problems seem to be a combination of (1) the way backblaze does not immediately reflect prune actions in restic and (2) some [hash1]-restic-temp-[hash2] files being left around in the data directory of the repository that prune isn’t able to clean up.
#1 I can deal with by choosing another provider if I don’t like that behavior.
#2 Isn’t really a problem, per se, as much as it is an annoyance. The fact that restic leaves behind ~60 MB of stale data per interrupted backup isn’t a ton in today’s day and age. But it has the potential to add up over time. I just wanted to know if those restic-temp files can be cleaned up manually (of course, doing so when no backups are running and nothing is actively writing to the repository).
Thank you for the link to the Design Doc. I did read through that earlier this afternoon and it was helpful to understand some of what was going on behind the scenes.
It looks like they’re all safe to remove and perhaps will be cleaned up in a future release.
That is very interesting. I just searched my 2.2 TB of backups here at home and didn’t find a single tmp or temp file. I’m using rest-server as a backend fwiw.
