Forget policy not working as excpected

Getting Help
1

Hello

I need help to fix one of my forget commands and clean up my snapshots according to my retention policy.

Here is my retention policy:

  • Keep all backups tagged “daily” for 15 consecutive days.
  • Beyond these 15 days, keep all backups tagged “bi-monthly” for 6 months. For explanation, these are backups made on the 02nd and 16th of each month.

Here is an overview of my snapshots:

ID        Time                 Host      Tags       Size
------------------------------------------------------------------------
a2e7bdfb  2023-03-16 10:41:35  hostname  bi-monthly
4a17d807  2023-04-02 01:00:02  hostname  bi-monthly
dc09fe1e  2023-04-16 01:00:04  hostname  bi-monthly
aeebffb2  2023-05-02 01:00:01  hostname  bi-monthly
3887e1d7  2023-05-16 01:11:02  hostname  bi-monthly
4cbad374  2023-06-02 01:09:01  hostname  bi-monthly
1ebab9c6  2023-06-16 01:06:02  hostname  bi-monthly
955a07d2  2023-07-02 01:08:02  hostname  bi-monthly
eb73d17a  2023-07-16 01:02:02  hostname  bi-monthly
e37b4edb  2023-08-02 01:05:02  hostname  bi-monthly
ac338b39  2023-08-16 01:09:01  hostname  bi-monthly
430efeaa  2023-09-16 01:04:01  hostname  bi-monthly
53937330  2023-10-02 01:03:02  hostname  bi-monthly
dfef820d  2023-10-16 01:08:02  hostname  bi-monthly
337ce00d  2023-11-02 01:13:02  hostname  bi-monthly
684a0c3c  2023-11-16 01:12:02  hostname  bi-monthly
b0099f6f  2023-12-02 01:06:02  hostname  bi-monthly
4b8defa0  2023-12-16 01:01:01  hostname  bi-monthly
16168401  2024-01-02 01:03:02  hostname  bi-monthly
e1330f48  2024-01-16 01:01:01  hostname  bi-monthly
5d4e8d21  2024-02-02 01:15:01  hostname  bi-monthly
21814691  2024-02-16 01:02:02  hostname  bi-monthly
f060c025  2024-03-02 01:05:02  hostname  bi-monthly
c89d7ff3  2024-03-16 01:09:02  hostname  bi-monthly
44b66b07  2024-04-02 01:09:01  hostname  bi-monthly
05144bdf  2024-04-16 01:04:02  hostname  bi-monthly
e56c675c  2024-05-02 01:10:02  hostname  bi-monthly
5bdfae2e  2024-05-16 01:01:01  hostname  bi-monthly
ce9826e7  2024-06-02 01:11:02  hostname  bi-monthly
ef86619e  2024-06-16 01:15:01  hostname  bi-monthly
8f9659b8  2024-07-02 01:09:01  hostname  bi-monthly
a7214e39  2024-07-16 01:13:02  hostname  bi-monthly
45c6f78e  2024-08-02 15:18:06  hostname  bi-monthly  5.138 GiB
c29ec2d4  2024-08-16 01:11:01  hostname  bi-monthly  5.117 GiB
59449a0b  2024-09-02 01:03:01  hostname  bi-monthly  5.093 GiB
8f336cf4  2024-09-16 01:03:02  hostname  bi-monthly  5.719 GiB
60f6208b  2024-10-02 01:02:01  hostname  bi-monthly  5.668 GiB
991e38b2  2024-10-16 01:02:02  hostname  bi-monthly  5.670 GiB
9f15c067  2024-10-20 01:09:23  hostname  daily       5.086 GiB
ec6591de  2024-10-21 01:14:02  hostname  daily       5.034 GiB
9a21dde5  2024-10-22 01:09:01  hostname  daily       5.654 GiB
8bf53338  2024-10-23 01:09:02  hostname  daily       5.658 GiB
ecae5f1d  2024-10-24 01:10:02  hostname  daily       5.667 GiB
ac975e5f  2024-10-25 01:06:02  hostname  daily       5.673 GiB
1095479f  2024-10-26 01:03:02  hostname  daily       5.682 GiB
238ae7b7  2024-10-27 01:02:01  hostname  daily       5.690 GiB
96761f69  2024-10-28 15:06:01  hostname  daily       5.653 GiB
c5967fe1  2024-10-29 01:08:02  hostname  daily       5.656 GiB
b9e92f62  2024-10-30 01:12:01  hostname  daily       5.231 GiB
c2481e2c  2024-10-31 01:14:01  hostname  daily       5.173 GiB
4ab09eb5  2024-11-01 01:02:22  hostname  daily       5.176 GiB
fc9c0028  2024-11-02 01:04:01  hostname  bi-monthly  5.184 GiB
062eb378  2024-11-03 01:05:22  hostname  daily       5.183 GiB
2504a0dc  2024-11-04 01:11:02  hostname  daily       5.180 GiB

Currently, I use 2 forget commands but only one of them works as expected:

/usr/bin/restic forget --tag daily --keep-daily 15

=> works as expected, as can be seen in the snapshots above, only 15 “daily” snapshots are kept.

/usr/bin/restic forget --tag bi-monthly --keep-daily 180

=> does not work as expected, as can be seen in the snapshots above, “bi-monthly” snapshots from 2023 are still present even though it is November 2024…

What do I miss?

Thanks for your help

2

It does work exactly as expected - you want to keep 180 daily snapshots with this tag. At least it is what you ask for. It does not matter what time period - 180 days or 5 years - last 180 daily snapshots will be kept.

From your description it sounds like you want to use --keep-within-daily 180 and not --keep-daily 180.

Run some tests. Use --dry-run flag when experimenting.

Check docs for details,

1 Like
3

Well maybe, but I found that part of the documentation is really hard to understand.

  • --keep-daily n for the last n days which have one or more snapshots, keep only the most recent one for each day.
    => I was expecting: for the last 180 days (from today to 08/05/2024 then), keep only the most recent one for each day.

I tried with --keep-within-daily 180d but this has no effect (nothing more is removed).

There are so many options but I feel like what I want to do is not possible.

Thanks for your help anyway

4

/usr/bin/restic forget --tag "bi-monthly" --keep-within-daily 180d

should remove any snapshots older than 6 months as you request, as @kapitainsky said. Also -keep-within-weekly 180d should work.
Do you have different paths in those backups? Then adding --group-by tag would help to treat them as one kind of backup.

5

I agree. I only cracked it experimenting with --dry-run:slight_smile:

If you think you have idea how to make docs better please submit PR.

6

You skipped “which have one or more snapshots” part from docs. Without it meaning is different indeed.

Again I agree 100% that it is not idiot proof explanation in docs. And by idiot I mean myself as I struggle to understand it when reading.

7

What about:

Instead of:

--keep-daily n for the last n days which have one or more snapshots, keep only the most recent one for each day.

maybe it should be rather:

--keep-daily n - keep the last n daily snapshots. Only the most recent snapshot for each day will be kept. Days without any snapshots are not counted towards n.

What others think?

1 Like
8

Btw, I always read the --keep-within-* duration options as “keep * within duration” instead of “keep within * duration”. Unfortunately, they are not defined in this order. E.g. “keep daily (snapshots) within 31d(ays)” = --keep-daily-within 31d.