Curious case of the Forget policy

Getting Help
1

Hi,
I would like to check if there is a possibility to merge snapshots in the same repository? I am having some curious effects using “forget” because it recognises different patterns (so to speak)

My scenario is as follows:

  • Initial backups was done individually on a folder by folder basis with the assumption that I could have a few folders fully backed up before proceeding with the next within the time I have available without running an entire backup and wait for the process to complete.

  • Subsequent backups where done by selecting all backup sources at once once I found that it is easier to browse through the files using FUSE. This would also mean that the backups would deduplicate files and folders already backed up hence not drastically increasing my repo size.

  • All was good in the backup front, however when I want to prune things work differently from what I assumed it would be. The forget policy keeps taking each so called source signature as an individual backup. As in, if I say delete all snapshots in the last 30 days or keep-last 30 it lists the last 30 snapshots that belong to each source signature/pattern.

E.g. Day 1 I backed up D:\Work_Folder, Day 2 I backed up My Documents folder, Day 5 I backed up both D:\Work_Folder & My Documents folder, this is taken as 3 different signatures. So if I continue to backup D:\Work_Folder & My Documents for 40 days and then set a forget keep-within 0y1m flag I assumed that last 30 days of snapshots in my repo will be preserved deleting older ones. However, what happens is that backups done on Day 1 and Day 2 are preserved supposedly they meet the criteria even though they where done 40 days ago and the backups done with the pattern done on Day 5 works as expected with those older than 30 days being removed.

I would assume that since I backed up from the same sources using the same machine the subsequent backups should have indexed the files and folders in the latest snapshot, so why does the forget policy not apply for the entire repo instead only applies for a particular pattern?

Please find the output from the dry run below:

PS C:\Users\User> restic -r W:\restic\ forget --keep-within 0y1m --prune --dry-run
enter password for repository:
repository a6a81653 opened successfully, password is correct
Applying Policy: all snapshots within 1m of the newest
keep 22 snapshots:
ID        Time                 Host        Tags        Reasons    Paths
------------------------------------------------------------------------------------------
707e4723  2019-04-23 15:23:51  Toblerone               within 1m  C:\Users\User\Documents
                                                                  C:\Users\User\Downloads
                                                                  C:\Users\User\Pictures
                                                                  D:\

019ef9b0  2019-04-24 10:30:01  Toblerone               within 1m  C:\Users\User\Documents
                                                                  C:\Users\User\Downloads
                                                                  C:\Users\User\Pictures
                                                                  D:\

cdd72eeb  2019-04-25 15:45:54  Toblerone               within 1m  C:\Users\User\Documents
                                                                  C:\Users\User\Downloads
                                                                  C:\Users\User\Pictures
                                                                  D:\

bfb5a87f  2019-04-26 13:24:55  Toblerone               within 1m  C:\Users\User\Documents
                                                                  C:\Users\User\Downloads
                                                                  C:\Users\User\Pictures
                                                                  D:\

351fb28f  2019-04-29 09:55:17  Toblerone               within 1m  C:\Users\User\Documents
                                                                  C:\Users\User\Downloads
                                                                  C:\Users\User\Pictures
                                                                  D:\

e10b2206  2019-04-29 10:30:01  Toblerone               within 1m  C:\Users\User\Documents
                                                                  C:\Users\User\Downloads
                                                                  C:\Users\User\Pictures
                                                                  D:\

d1a5e093  2019-04-30 14:14:54  Toblerone               within 1m  C:\Users\User\Documents
                                                                  C:\Users\User\Downloads
                                                                  C:\Users\User\Pictures
                                                                  D:\

4f30bec7  2019-05-02 09:49:10  Toblerone               within 1m  C:\Users\User\Documents
                                                                  C:\Users\User\Downloads
                                                                  C:\Users\User\Pictures
                                                                  D:\

9198acd2  2019-05-02 10:30:00  Toblerone               within 1m  C:\Users\User\Documents
                                                                  C:\Users\User\Downloads
                                                                  C:\Users\User\Pictures
                                                                  D:\

3e0dd2a2  2019-05-03 17:16:31  Toblerone               within 1m  C:\Users\User\Documents
                                                                  C:\Users\User\Downloads
                                                                  C:\Users\User\Pictures
                                                                  D:\

ec592370  2019-05-06 09:46:45  Toblerone               within 1m  C:\Users\User\Documents
                                                                  C:\Users\User\Downloads
                                                                  C:\Users\User\Pictures
                                                                  D:\

cd019f73  2019-05-06 10:30:01  Toblerone               within 1m  C:\Users\User\Documents
                                                                  C:\Users\User\Downloads
                                                                  C:\Users\User\Pictures
                                                                  D:\

29acc393  2019-05-07 10:30:06  Toblerone               within 1m  C:\Users\User\Documents
                                                                  C:\Users\User\Downloads
                                                                  C:\Users\User\Pictures
                                                                  D:\

8583575b  2019-05-08 10:30:03  Toblerone               within 1m  C:\Users\User\Documents
                                                                  C:\Users\User\Downloads
                                                                  C:\Users\User\Pictures
                                                                  D:\

77e2f4de  2019-05-09 10:30:03  Toblerone               within 1m  C:\Users\User\Documents
                                                                  C:\Users\User\Downloads
                                                                  C:\Users\User\Pictures
                                                                  D:\

79035c32  2019-05-10 10:30:09  Toblerone               within 1m  C:\Users\User\Documents
                                                                  C:\Users\User\Downloads
                                                                  C:\Users\User\Pictures
                                                                  D:\

86783039  2019-05-14 10:30:04  Toblerone               within 1m  C:\Users\User\Documents
                                                                  C:\Users\User\Downloads
                                                                  C:\Users\User\Pictures
                                                                  D:\

9fc58c7c  2019-05-15 10:30:04  Toblerone               within 1m  C:\Users\User\Documents
                                                                  C:\Users\User\Downloads
                                                                  C:\Users\User\Pictures
                                                                  D:\

11fcb6c3  2019-05-17 09:57:15  Toblerone               within 1m  C:\Users\User\Documents
                                                                  C:\Users\User\Downloads
                                                                  C:\Users\User\Pictures
                                                                  D:\

77845734  2019-05-21 10:30:01  Toblerone               within 1m  C:\Users\User\Documents
                                                                  C:\Users\User\Downloads
                                                                  C:\Users\User\Pictures
                                                                  D:\

65a677a3  2019-05-22 10:30:16  Toblerone               within 1m  C:\Users\User\Documents
                                                                  C:\Users\User\Downloads
                                                                  C:\Users\User\Pictures
                                                                  D:\

f73b541b  2019-05-23 10:30:04  Toblerone               within 1m  C:\Users\User\Documents
                                                                  C:\Users\User\Downloads
                                                                  C:\Users\User\Pictures
                                                                  D:\
------------------------------------------------------------------------------------------
22 snapshots

remove 22 snapshots:
ID        Time                 Host        Tags        Paths
-------------------------------------------------------------------------------
c9faa05c  2019-02-28 12:23:31  Toblerone   FEB,END     C:\Users\User\Documents
                                                       C:\Users\User\Downloads
                                                       C:\Users\User\Pictures
                                                       D:\

e3ffd381  2019-03-01 15:10:57  Toblerone   MAR,START   C:\Users\User\Documents
                                                       C:\Users\User\Downloads
                                                       C:\Users\User\Pictures
                                                       D:\

afb1c018  2019-03-11 10:34:53  Toblerone   MAR,MID     C:\Users\User\Documents
                                                       C:\Users\User\Downloads
                                                       C:\Users\User\Pictures
                                                       D:\

26898663  2019-03-12 14:47:05  Toblerone   MAR,MID     C:\Users\User\Documents
                                                       C:\Users\User\Downloads
                                                       C:\Users\User\Pictures
                                                       D:\

90400685  2019-03-13 16:53:41  Toblerone   MAR,MID     C:\Users\User\Documents
                                                       C:\Users\User\Downloads
                                                       C:\Users\User\Pictures
                                                       D:\

fa2f2685  2019-03-19 16:30:26  Toblerone   MAR,MID     C:\Users\User\Documents
                                                       C:\Users\User\Downloads
                                                       C:\Users\User\Pictures
                                                       D:\

e4a084b3  2019-03-28 12:11:38  Toblerone               C:\Users\User\Documents
                                                       C:\Users\User\Downloads
                                                       C:\Users\User\Pictures
                                                       D:\

ea18e631  2019-04-02 10:30:01  Toblerone               C:\Users\User\Documents
                                                       C:\Users\User\Downloads
                                                       C:\Users\User\Pictures
                                                       D:\

4d978c64  2019-04-03 10:30:02  Toblerone               C:\Users\User\Documents
                                                       C:\Users\User\Downloads
                                                       C:\Users\User\Pictures
                                                       D:\

8f669003  2019-04-04 10:30:02  Toblerone               C:\Users\User\Documents
                                                       C:\Users\User\Downloads
                                                       C:\Users\User\Pictures
                                                       D:\

e14acf8d  2019-04-05 14:07:42  Toblerone               C:\Users\User\Documents
                                                       C:\Users\User\Downloads
                                                       C:\Users\User\Pictures
                                                       D:\

e49a19ec  2019-04-08 10:30:00  Toblerone               C:\Users\User\Documents
                                                       C:\Users\User\Downloads
                                                       C:\Users\User\Pictures
                                                       D:\

c948dcfa  2019-04-09 12:43:09  Toblerone               C:\Users\User\Documents
                                                       C:\Users\User\Downloads
                                                       C:\Users\User\Pictures
                                                       D:\

d9d344df  2019-04-10 10:30:03  Toblerone               C:\Users\User\Documents
                                                       C:\Users\User\Downloads
                                                       C:\Users\User\Pictures
                                                       D:\

6911cf59  2019-04-11 10:30:04  Toblerone               C:\Users\User\Documents
                                                       C:\Users\User\Downloads
                                                       C:\Users\User\Pictures
                                                       D:\

ffd1ab70  2019-04-12 10:30:04  Toblerone               C:\Users\User\Documents
                                                       C:\Users\User\Downloads
                                                       C:\Users\User\Pictures
                                                       D:\

15a63807  2019-04-15 09:41:51  Toblerone               C:\Users\User\Documents
                                                       C:\Users\User\Downloads
                                                       C:\Users\User\Pictures
                                                       D:\

6fba037c  2019-04-15 10:30:01  Toblerone               C:\Users\User\Documents
                                                       C:\Users\User\Downloads
                                                       C:\Users\User\Pictures
                                                       D:\

b1782998  2019-04-16 12:40:12  Toblerone               C:\Users\User\Documents
                                                       C:\Users\User\Downloads
                                                       C:\Users\User\Pictures
                                                       D:\

db51cd43  2019-04-17 10:30:04  Toblerone               C:\Users\User\Documents
                                                       C:\Users\User\Downloads
                                                       C:\Users\User\Pictures
                                                       D:\

29cfa4b8  2019-04-18 10:30:03  Toblerone               C:\Users\User\Documents
                                                       C:\Users\User\Downloads
                                                       C:\Users\User\Pictures
                                                       D:\

cf5a4fc7  2019-04-22 09:52:12  Toblerone               C:\Users\User\Documents
                                                       C:\Users\User\Downloads
                                                       C:\Users\User\Pictures
                                                       D:\
-------------------------------------------------------------------------------
22 snapshots

keep 1 snapshots:
ID        Time                 Host        Tags        Reasons    Paths
-----------------------------------------------------------------------
464d7afb  2019-02-22 10:05:07  Toblerone   FEB,END     within 1m  d:\
-----------------------------------------------------------------------
1 snapshots

keep 1 snapshots:
ID        Time                 Host        Tags        Reasons    Paths
------------------------------------------------------------------------------------------
7e1f669e  2019-02-26 16:13:36  Toblerone   FEB,END     within 1m  C:\Users\User\Documents
------------------------------------------------------------------------------------------
1 snapshots

22 snapshots have been removed, running prune

=====================================================

2

By default restic groups by host and path ( --group-by host,paths). All of the snapshots above have the same hostname and path. That’s why restic considers these snapshots as 1 ‘dataset’. All of the snapshots which restic wants to keep are within 1 month of the latest snapshot (2019-05-23). So this looks good to me.

These snapshots have a different path, hence restic considers them as another ‘dataset’. Both of them are within 1 month of the latest snapshot (restic will always keep atleast 1 snapshot). So this looks good to me aswell.

3

Hi,
Thank you for this info and understood on the above.

Is it safe to assume that since the initial datasets which has the paths already referenced by the later backup datasets can be removed manually without any data loss?

Thank you.

4

Yes. Or you can use --group-by host to tell restic that you don’t want it to consider paths at all; then all the snapshots will be part of the same group. Right now, the two snapshots with a different path set are in their own group.

1 Like
5

Thank you and noted on the above, that information would be very useful.