Can I use S3 provider's lifecycle policy instead of pruning?

Getting Help
1

Hey all,

I’m using volsync in Kubernetes with the restic datamovers, to backup a large amount of volumes to R2/B2 buckets for disaster recovery purposes.

I only care about recovering the last full backup in the event of a disaster, and so have configured a retention period of 2 days (the backups run daily). I’ve read about the difference between “forgetting” and “pruning”, but I’m not understanding the underlying mechanism behind pruning, and why it’s an i/o-heavy operation…

If (1) restic only ever writes to a file (object) once, and (2) I’m only retaining 2 daily backups, then couldn’t I just instruct B2/R2 to delete any file older than 3 days, as part of a bucket lifecycle policy? In this case, would I ever care about running pruning?

It seems like “easy mode”, compared to the idea of i/o-intensive “repacking”, so I’m wondering what I’m missing? :slight_smile:

Thanks!
D

1 Like
2

If your retention policy deletes files that haven’t been updated for a few days, your repository will be irretrievably (1) lost. Files in the Restic repository are never updated - only created or deleted.

You may have missed the part about deduplication! New snapshots only add what has changed / what is new compared to the previous snapshots. That’s why you can’t just delete old pack files.

About retention policies: When backing up to object storage, I configure the application key to be unable to delete anything. It can only “hide” files, and after a few weeks, hidden files are then finally deleted by the retention policy. This way, if I mess up my remote repository somehow, I can always “go back in time” and recover everything, including forgotten and pruned snapshots.

Useful link to the repository design

(1) Learned a new word today. I hope it makes sense in english. :upside_down_face:

2 Likes
3

Using S3 lifecycle policies to remove old files, completely and irrecoverably breaks the repository. The oldest files in a repository are usually the config and key files. Without those the data in the repository is just encrypted gibberish without any possibility to restore data.

2 Likes