Newbie question: Problem with forgetting/pruning changing subfolders

[ebola]

Hi there,
I use restic to backup my mailserver. The server uses “Maildir” for all linux users. So my backup-command looks like this:

restic -r sftp:username@backupserver:/media/pi/SanFit/restic_mail backup /home/*/Maildir/**

Backup as such works. But when it comes to pruning, this won’t work.
I tried:
restic -r sftp:username@backupserver:/media/pi/SanFit/restic_mail forget --keep-daily 7 --keep-weekly 4 --keep-monthly 12

One subfolder (/home/email/Maildir/dovecot-uidvalidity.6089abc3) seems to be changing the last 8 characters. And restic seems to create a different brach for each snapshot with a different subfolder. Thus, even if I do “–keep-last 1” it wants to keep a lot of snapshots. (One with each of the different subfolders)

Any way around this?

Thanks in advance!

[arkadi]

I was reading a thread lately that I think is relevant to your problem.

Restic incremental backup report confusing

The --group-by option is for when you want to delete snapshots, not create them. As long as you have the disk space, there isn’t any rush to think about this now; get comfortable with the restic backup process first, and then start learning about snapshot deletion and --group-by here. The challenge you’re facing right now is only about optimizing your backup process. It needs some additional info if you want it to run as fast as possible. The presence of a parent snapshot speeds-up a backup be…

[Nev]

A clarifying point on terminology:

And restic seems to create a different brach for each snapshot with a different subfolder.

Yes, restic isn’t doing any “branching” at backup time; all sounds fine there. But when it’s time to forget, restic’s auto-grouping isn’t doing what you desire. So manually tell it how you want the snapshots grouped. And run it first with the --dry-run flag to be sure it’ll do what you want.

In the thread referenced above by @arkadi, @fd0 had the pertinent advice:

@fd0

Please be aware that the retention policies applied by restic forget will group snapshots according to the list of directories they contain, so right now you will end up with two groups (for the two lists of dirs). If you don’t want that, you need to tell restic to only group by hostname using --group-by host (the default is --group-by host,paths).

[Nev]

Just a side note.

I think you’d still be backing-up the same data with a path of /home/*/Maildir ? Then forget's default behaviour wouldn’t identify so many groups (and backup would more-often identify a parent snapshot to speed-up that operation, too). But a new group would still be detected when your list of users changed, so you’d still need the manual --group-by eventually.

[torfason]

@Nev

I think you’d still be backing-up the same data with a path of /home/*/Maildir ? Then forget’s default behaviour wouldn’t identify so many groups …

Yes, this is surely the issue. What @Nev is talking about, and which may be worth emphasising a bit, is that when you backup using wildcards, such as /home/*/Maildir/**, restic does not store that string as the snapshot path, but instead it stores a list of each matching path. So the grouping (which includes path by default) will be all messed up.

Omitting the last /** means that the backup path is the Maildir directory – and restic will back up everything found under that directory unless it is specifically excluded, and changes within the Maildir will not affect which paths restic sees.

You probably want to check out the --group-by parameter anyway, to be prepared (and perhaps to clean up the snapshots you already created), but this change should make future management much cleaner.

And if you are managing a mail server with multiple actual users, I would seriously consider backing up in a loop to do each user separately. That would take care of the issue that Nev points out, which is that a new group would be created any time a new user is added. And you would get the added benefit that the retention policy would really target each user, keeping the correct interval snapshots for each user and allowing you to forget all snapshots for a given user if that were necessary for some reason.

[ebola]

Thanks everybody for the great feedback! I tried it out with “/home//Maildir" instead of "/home//Maildir/**”. Looks much better. Now i just get paths for my users. As I actually don’t have any fluctuation of users (small scale server…) this will probably do fine for me.
I’ll give it a try over a longer period of time, to see, how it pans out.
Thanks again!