Auto-snapshots / smart backup rentention

Getting Help
1

Is there anyone who has written a script similar to zfstools auto-snapshot for zfs ?

TLDR; it makes keeping snapshots for intervals and deleting old copies a cinch.

Duplicati offers this feature in the form of “smart backup retention”

Over time backups will be deleted automatically. There will remain one backup for each of the last 7 days, each of the last 4 weeks, each of the last 12 months. There will always be at least one remaining backup

2

So if by that you mean that the script itself will recognize if there are, for example, more than 7 backups in one day then it will run the forget and prune commands applying your policies, I haven’t seen any script like this for Restic. It’s not the same but I use this script. You can add a custom “policy” for when you want to run forget and prune according to your choice. Author says it is 7 days by default but you can change it. So if you run the script hourly, for example, then it will run hourly and keep all snapshots. Then at the exact 7th day after your last “cleanup” run it will run forget and prune again and reset the date for another 7 days. It is not as smart as the duplicati script you mention but for me it works. Besides it has a couple of interesting options.

Edit:

By the way, using this same example to keep 7 snapshots daily and running your script hourly will make the forget a prune process to run every day to forget snapshots from the day before and that is why I prefer to do it weekly because right now the prune process could take a lot of time if you have a lot of data or your destination is a remote location. I think they’re trying to improve this but it could take a while.

1 Like
3

yes, like that. Interesting about the prune. If no one has such a script, I’ll likely write one.

And sounds like I should have the prune happen only weekly instead of with every daily backup

4

These are another two:

3 Likes
5

Ah, those look useful. Thank you :slight_smile:

1 Like
6

Just for completion of this post, the solution based on those scripts was that I am running the command below once per week in a cronjob

restic forget --keep-daily 7 --keep-weekly 5 --keep-monthly 12
7

Just wanted to add that this command removes some snapshots from the list but doesn’t actually remove any data. You’d need to run restic forget --prune --keep-daily 7 --keep-weekly 5 --keep-monthly 12 to remove unneeded data and free up some space. Alternatively you can run restic prune after restic forget.

1 Like
8

I ended up getting very specific with my backups. Maybe this can help someone else.

The way this works is that a monthly backup has weekly rolled into it, and a weekly backup has a daily backup rolled into it.

I’m using a Makefile:

build:
    @echo building includes...
    @rm -rf ${RESTIC_INCLUDE}/*
    @cp src/daily ${RESTIC_INCLUDE}/daily
    @cat src/daily src/weekly > ${RESTIC_INCLUDE}/weekly
    @cat src/daily src/weekly src/monthly > ${RESTIC_INCLUDE}/monthly

crontab:
    crontab -l | restic backup --stdin --tag crontab

daily: build
    restic backup --files-from ${RESTIC_INCLUDE}/daily --tag daily

weekly: build
    restic backup --files-from ${RESTIC_INCLUDE}/weekly --tag weekly

monthly: build
    restic-make
    restic backup --files-from ${RESTIC_INCLUDE}/monthly --tag monthly

forget:
    restic forget --keep-daily 7                 --tag daily -g tags
    restic forget                --keep-weekly 4 --tag weekly -g tags
    restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 12 -g tags


space:
    restic stats --mode raw-data

There’s a source directory that builds the files for daily, weekly, monthly and then the various backups are tagged as necessary.

$ ls $RESTIC_HOME/src/
daily  monthly  weekly

Then I run this via crontab:

15 0 * * *   make -C $RESTIC_HOME build daily
30 2 * * *   make -C $RESTIC_HOME forget
0  2 * * sun make -C $RESTIC_HOME build weekly
0  3 * * sun make -C $RESTIC_HOME prune
0  4 1 * *   make -C $RESTIC_HOME build monthly

The environmental variables being set are excluded. I actually set them in my shell. They can be set in the crontab, of course.

This keeps all monthly backups for 1 year, all weekly for 4 weeks, and all daily for 7 days. Since all the backups overlap, everything is backed up for one year.

I hope this helps someone else

1 Like