Hi everyone. Apologies if this has been covered before - I might have missed a previous discussion or some details.
While browsing the forum, I came across an idea regarding backup retention. My AI assistant pointed me to Issue#3195, but I believe my suggestion is slightly different. I think it would be very useful to have a “lock” feature for specific snapshots.
There are cases where I need to preserve a particular backup without altering the global backup plan or editing scripts. Essentially, I want to manually ‘lock’ a snapshot to prevent it from being deleted until I explicitly unlock it. While S3 offers deletion policies, I’m looking for a more direct, manual way to ensure specific snapshots are kept regardless of automated cleanup rules.
I presume you now about forget --keep-tag which you could have always applied to your forget commands with a specific tag you chose to represent “locked” snapshots, e.g. forget --keep-tag locked? Any snapshot you put that tag on will then be kept regardless of the rest of the policy.
I see your point about --keep-tag, and that’s what I’m using now. But for me, it feels like a workaround rather than a solid safety feature.
The real issue is ‘Silent Success’. If I make a typo in a tag within my cron script, forget will just happily run, find no matches, and delete the snapshot I was trying to protect. If we had an internal lock state, a typo in the ID would cause an immediate error, stopping the process. That’s a huge difference in peace of mind.
To give you a real-world example: whenever I need to perform a data restore, my first step is to take a ‘pre-restore’ backup of the current (broken) state. It’s a vital safety net in case I need to check something from the version I’m about to overwrite. In the middle of a stressful recovery, I want to just lock that specific snapshot and know it’s safe. I don’t want to worry about a nightly cron job pruning it because I forgot to update the scripts or messed up a tag while under pressure.
Also, I believe that if I lock a snapshot, it should be ‘invisible’ to the retention policy. If I want to keep the last 5 backups, I want exactly 5 fresh ones, plus whatever n snapshots I’ve manually locked for the long term. Using tags makes this math messy because they ‘steal’ slots from the regular rotation.
I’m basically looking for a way to make the snapshot ‘protect itself’ from the inside, so I don’t have to rely on a fragile external script to remember which tag is which.
Another option is to always use --group-by host,tags. I now use this in all of my retention policy scripts for several reasons. For example, it’s common for a logical set of backups to have paths that change over time, but the default --group-by host,paths would treat the snapshots with a different set of paths as a different backup set. And, as you’ve found, this makes it too easy for a forgotten --keep-tag to remove snapshots you wanted to save (I use keep for such snapshots).
With --group-by host,tags, it’s extremely unlikely that your “locked” snapshot in this example would be removed. This is not to say that your proposal doesn’t have value, but rather this is a suggestion for something you can do right now that will make it far less likely to accidentally delete a snapshot – you would effectively have two different layers of protection instead of one.
@Forrend noted but you can just as easily make a ‘typo’ with your o-so-important-snapshot-to-be-locked! What you are asking is basically archival, in addition to backups.
In this thread you were already given a few good suggestions on how to deal with the situation.
You’re mistaken. If I make a typo in the snapshot ID, it will simply return an error saying the snapshot wasn’t found. However, if I make a typo in the tag exclusion, the script won’t find the match and will skip the snapshot, leading to its deletion. In any case, these are completely different things. Blocking is a manual process, whereas skipping a specific tag requires modifying the cleanup scripts, which is much more inconvenient.
From this thread it’s understood that there are ways to use a combination of tags and the --group-by host,tags in the forget command to achieve the required behaviour to prevent ‘silent success’ deletion of important snapshots.
IMO what @Forrend is asking for would be a notable feature to have. Some of the other use cases I can think of is that this can be extended to providing immutability to snapshots and associated data in storage backends that support it.
I am also a user of --keep-tag and it works fine, but AFAIU these are 2 different problems.
–keep-tag prevents forgetting snapshots voluntarily, which is a good solution to provide to the clients (e.g. “you can prevent automatic removal/cleanup of your selected snapshots if you tag them ‘persistent’”)
Main issue is about security (or I misunderstood): You want a way to prevent users remove a snapshot within the boundaries of Restic.
The problem is: Even if this is implemented, it will be a “soft lock” with current design. Restic does not manage the access to the actual files in the repo and as you probably know everything is just a file within a reachable target, a.k.a. repo. Forgetting a snapshot is just removing the reference file inside snapshots/ directory of the repository, nothing else.
I am not sure what is the right approach here. It sure sounds helpful to remind a user by preventing accidental forgets: “hey you should not delete this one”. But in reality we still need the access controls to the repo with the current way of things, and I am sure some users would confuse any soft-locking mechanism as legit security feature, even if it’s documented correctly.
I completely agree that since restic is file-based, any internal lock is not a replacement for filesystem-level immutability or access control.
However, we should distinguish between Security (protection against malice) and Safety (protection against accidents). Most operations are automated via crontabs. A “soft lock” stored in the snapshot metadata would act as a vital safety catch. It tells the command: “I don’t care what your policy says, skip this ID”.
@Forrend you can make a typo and copy+paste the previous snapshot id to be locked, or you can make a typo in the path and have an empty or incomplete backup etc etc etc.
Whereever one does things manually we have to accept that mistakes can be made and these can not always be caught by checks and (endless) safely mechanisms.
In your real-world example if you want to deal with it manually then I would invoke restic backup to a new repo or simply use copy/tar/rsync/rclone whatever works on your platform. No risk for pruning, no need to change your cron scripts. When done with the data restore you can remove your manual backup and be done.
But lets agree to disagree (and i’ll leave it at this). In software there are many ways to do things and as many preferences and opinions. 8-)
As @Forrend correctly pointed out, there are two things:
Support within restic to not remove specific snapshot based by information within the snapshotfile independent from any forget policy. I personally would call this delete protection as lock is typically associated with a storage backend capability. This is not implemented in restic but actually quite easy to add. (And I know some people dislike me pointing to rustic, but it does implement this functionality using a delete field in the snapshot file which allows to be set as Never )
Support within restic to use and set some locking capabilities within the backend, e.g. by an ACL-based access restriction or some compliance-supporting locking capabilities. I would see this on-top of what I called delete protection (if restic knows, that a user wants this snapshot be extra-protected by the storage backend, it makes no sense to try to remove it as this would anyway just give a backend error), but actually it is more complex: To ensure that not only the snapshot file is locked but also the content that snapshot references, the backend should lock the needed pack files also. So, this would mean to handle locks (in the sense of storage backend locks) also for pack files which is more involved as this requires handling of this withing prune.