I have a repo with a lot of hosts (>700). Each of these hosts has possibly different retention periods, which is kept in a database. I’m currently running a loop, which runs forget for each of these hosts, with locking/unlocking this takes a lot of time.
Do you have any suggestions to do this faster? I thought I can manually lock the repo and read stuff under /snapshots via cat, but that would be re-implementing the retention. It’d be great if we could chain multiple forget commands with their respective --keep-* params into single command, but not sure if technically possible.
A forget can be given multiple --keep- switches. “Multiple policies will be ORed together so as to be as inclusive as possible for keeping snapshots.” from the docs.
I know that the prune performance has been improved in 0.12.0 and further since, but not sure if this also covers forget performance.
I’m already utilizing this, keep 1 monthly, 1 weekly and 2 daily for a host etc. But this doesn’t help with hosts which has to be applied different policies. Each host needs to be given one by one.
One idea would be grouping same policies and giving them for multiple hosts at once. But I need to test it, since I am not sure how would --group-by host affects the retention
restic 0.12.0 should be already faster with forget as the reading of snapshot files is parallelized IIRC.
You could run restic snapshots --json and parse the JSON array with an algorithm that evaluates all your forget policies at once (e.g. group the array by host, sort each group by date and filter out the snapshots to delete) to yield the list of all snapshot IDs to forget and then run restic forget <ID-list>.
The problem is, however, that restic forget is not yet optimized for a long list of given IDs to remove (it will list the available snapshots for each given ID in order to find that ID - such that you are able to specify only the first characters of the IDs). So if this actual forget command is still too slow, I would advise you to directly remove the snapshot files from the backend in /snapshots/<ID>. This is also exactly what the forget command does.
EDIT: Just realized you don’t want to reimplement the forget command. This is what I effectively suggested
