Restic "Forget" Confusion

Getting Help
1

So I ran:

restic forget --group-by host --prune --keep-within 30d --keep-daily 90 --keep-weekly 26 --keep-monthly 12

Today is Sept 1st. 90 days ago was June 3rd.

But for some reason got:

repository 24344e18 opened successfully, password is correct
Applying Policy: keep the last 90 daily, 26 weekly, 12 monthly snapshots and all snapshots within 30d of the newest
keep 33 snapshots:
ID        Time                 Host                      Tags               Reasons           Paths
---------------------------------------------------------------------------------------------------------------------
a5ca6389  2019-05-28 07:11:54  akrabu-macbook-air.local                     daily snapshot    /Applications
                                                                                              /Library
                                                                                              /Users/akrabu
                                                                                              /Volumes/Drobo/!To Sort
                                                                                              /Volumes/Drobo/Pictures
                                                                                              /Volumes/Drobo/Text
                                                                                              /Volumes/Drobo/Utils

da6ce5ae  2019-05-29 19:59:59  akrabu-macbook-air.local                     daily snapshot    /Applications
                                                                            monthly snapshot  /Library
                                                                                              /Users/akrabu
                                                                                              /Volumes/Drobo/!To Sort
                                                                                              /Volumes/Drobo/Pictures
                                                                                              /Volumes/Drobo/Text
                                                                                              /Volumes/Drobo/Utils

01b43163  2019-06-01 13:11:39  akrabu-macbook-air.local                     daily snapshot    /Applications
                                                                            weekly snapshot   /Library
                                                                                              /Users/akrabu
                                                                                              /Volumes/Drobo/!To Sort
                                                                                              /Volumes/Drobo/Pictures
                                                                                              /Volumes/Drobo/Text
                                                                                              /Volumes/Drobo/Utils
<------------------------truncated------------------------>

I don’t understand why the first one was kept. It’s not the last monthly, nor the last weekly, nor in the last 90 days, nor in the last 30 days. It says “daily snapshot” though. I’m confused. Does --keep-daily mean keep the last 90 days worth of backups, not the last backup for the day for the last 90 days, as I interpreted it? :thinking:

2

I’m pretty sure that the way the daily flag works is: keep 90 snapshots on a one a day basis. You have 3 snapshots on 3 different days, which is less than 90, so they are all kept. If you use the -verbose flag you will see the rules being applied to each retained snapshot.

It will be interesting to see what the experts say about the combination of --keep-within 30d and --keep-daily 90. For instance, if you do two snapshots a day say over two months (30 days each), the first flag would keep 60, would the other flag count the ones kept as 30 or 60 when calculating the second flag; my guess would be 30, resulting with 60 from the last thirty days and 30 from the previous month.

3

Yes, @doscott is correct.
--keep-daily n means “keep the last n daily snapshots”, not “keep a daily snapshot for the last n days”.

In case of multiple policies — they don’t affect each other. They’re applied independently then ORed together, so the result is as inclusive/conservative as possible.

4

Wait, that doesn’t make sense:

–keep-last n never delete the n last (most recent) snapshots
–keep-hourly n for the last n hours in which a snapshot was made, keep only the last snapshot for each hour.
–keep-daily n for the last n days which have one or more snapshots, only keep the last one for that day.
–keep-weekly n for the last n weeks which have one or more snapshots, only keep the last one for that week.
–keep-monthly n for the last n months which have one or more snapshots, only keep the last one for that month.
–keep-yearly n for the last n years which have one or more snapshots, only keep the last one for that year.
–keep-tag keep all snapshots which have all tags specified by this option (can be specified multiple times).
–keep-within duration keep all snapshots which have been made within the duration of the latest snapshot. duration needs to be a number of years, months, days, and hours, e.g. 2y5m7d3h will keep all snapshots made in the two years, five months, seven days, and three hours before the latest snapshot.

Source: Removing backup snapshots — restic 0.16.3 documentation

--keep-daily 7 should keep backups ONE for the last 7 days, for any of those 7 days which happen to have a snapshot - aka one snapshot per day. If more than one snapshot was made in a day, only the latest would be kept, within the last 7 days.

–keep-monthly 7 for the last 7 months which have one or more snapshots, only keep the last one for that month.

--keep-within 7 should keep ALL snapshots within the last 7 days, no matter how many were made during those 7 days.

Take --keep-monthly 7 for instance. The phrasing is more clear for that one. It would only keep one per month, for the last 7 months. --keep-daily 7 should work the same way. It would only keep one per day, for the last 7 days.

Also, multiple policies just show together in the “Reasons (to keep)” column when you’re running the forget command. One snapshot may be kept due to ALL listed policies, or just one. They’re only removed if they match NONE.

I just don’t get why --keep-daily 90 would have been listed as a reason to keep a snapshot on 2019-05-28 07:11:54 when the cutoff for the last 90 days should have been 2019-06-03.

Double-checked my date math (adding one day, since it’s now the next day after this issue arose):

06%20AM

5

I think you’re reading a different meaning into the option descriptions. The description is a bit ambiguous.

–keep-daily n for the last n days which have one or more snapshots, only keep the last one for that day.

This can be read two ways. You are reading it like:

“For the last n consecutive days on the calendar, retain the last snapshot taken each day.”

You can also read it like this, which is how restic actually applies the logic.

“For the last n days that have a snapshot, retain the last snapshot taken each day.”

So if you say 30, it doesn’t mean the last 30 days on the calendar, it means the last 30 days for which a snapshot exists at all, and the days don’t have to be consecutive. This isn’t a bug, but the wording for the options could probably be tweaked to avoid the ambiguity.

I believe the wording “which have one or more snapshots” is the key phrase that attempts to explain the behavior, but you can still read the wording the other way.

2 Likes
6

Ahhh okay, well that explains that. I was starting to suspect that but wanted to see somebody confirm it. Thanks! :slight_smile:

7

But honestly, the whole retention policy can be really overwhelming.
This is a topic that comes up frequently here and on IRC I feel like.

I even made myself a test repo with hourly snapshots for a period over three years so I can test the forget policies and try to make some sense of it ha!
If anybody is interested, I’ve posted a link here in the forum: Script for creating a test repository
Don’t judge please :smiley:

1 Like
8

That’s exactly what restic does. The idea is that when somehow no new daily snapshots were made, but the policy was applied anyhow, you may end up with no snapshots at all. So restic only counts days which have a snapshot. It’s a safety feature.

I agree that the logic is not easy to understand. Imagine my confusion when I discovered that the retention policy was one of the most complex things within restic :wink:

@akrabu if you have any ideas on how to improve the description in the manual, please let us know! Feedback is very valuable for us!

2 Likes
9

Just found this thread and I think main update for the manual for me that would be useful is more of an example (possibly combined with some output) that would show this. Now that I read the actual interpretation of this line it makes sense to me and I’ve adjusted the config slightly, but before I also thought it was just removing the 13th snapshot while I set hourly retain to 12. Luckily due to the nature of the backups the actual effect on size etc is tiny but confusing nonetheless!