How often does restic delete data in a backup routine?

I imagine that restic stores some data in repositories that needs to be deleted occasionally, things like caches, etc (I’m not talking about my own blobs).

The reason I’m asking is because I’m considering to use restic with Wasabi through rclone, and Wasabi has a 90-day retention policy. Their FAQ page states:

Wasabi also has a minimum 90 day storage retention period that means stored objects that are deleted before they have been stored with Wasabi for 90 days will incur a pro-rated charge equal to the storage charge for the remaining days (see FAQ#6 for more details). This policy is comparable to the minimum storage charge period applicable to some of the AWS and other object storage services.

And I’d like to avoid these as much as possible as I would be running backup tasks very often. I’m not planning in prunning often.

Caches are stored on the local machine, not in the repository, so caches aren’t relevant here.

Lock files are the only repository files restic will delete when run as restic backup. In fact, nearly all commands will create and delete a lock file. These files are tiny, but depending on how many commands you run these might add up.

You can suppress the locking mechanism with --no-lock but then you are responsible for coordinating any operation that requires an exclusive lock and making sure that such an operation is the only thing running against the repository. Failure to do this can result in data loss, which is why the locking mechanism exists in the first place.

The only operations that delete repository files besides locks are (IIRC):

  • restic prune deletes files from data and index.
  • restic rebuild-index deletes files from index.
  • restic forget deletes files from snapshots. With --prune it also deletes files from data and index.
  • restic key remove deletes files from keys.
  • restic migrate might delete anything, depending on future changes to restic.
  • restic tag deletes files from snapshots.

Note that only forgetting snapshots older than 90 days is not a guarantee that restic prune won’t repack packs that were created more recently. In particular, if two concurrent backup operations duplicate some data (which can happen when they are both adding the same data during the same several-minute time period) then restic prune will discard all but one of the copies. This requires rewriting the pack(s) containing the duplicate(s).

2 Likes

Thanks for the in-depth answer.

You can suppress the locking mechanism with --no-lock but then you are responsible for coordinating any operation that requires an exclusive lock and making sure that such an operation is the only thing running against the repository.

This is good to know. So basically backing up from two computers at the same time is out of the question (and that’s OK, because my internet can’t handle that). I think the other commands you listed won’t be a problem because I don’t see myself doing them too often (except for rebuild-index, which for me the logical thing to do is to ensure nothing else is going with the repo when I run it).

I guess it will be worth it to understand Wasabi’s retention policy fully to see if running operations with no lock is worth it. It could be that the charges are minimal and I can just go with it.

Please be aware that not all operations support --no-lock, notably backup, prune, and forget do not.

Two operations like backup can run in parallel just fine, they only add new files to the repo (using pseudo-random names so they don’t collide).

2 Likes

For what it’s worth, I back up daily to wasabi and prune weekly with a 90 day retention. Daily additions are in the range of 50 KB to 10 GB. My wasabi storage is around 2.2 TB. Last month I had 9.1 GB-day timed deleted storage shown at a rate of $ 0.00016243 for a total of $0.

This month I pruned some things manually and I am showing 209 GB-day of timed deleted storage since April 27, with a cost of $0.03

So far there service has been reliable, works well with restic, and I have done a few restores of folders. It was nice not to have to worry about transfer charges.

1 Like

Backup takes a shared lock, not an exclusive lock. Two shared-lock operations can run concurrently with no issues.

1 Like