I think retention is not working as it should

Getting Help
1

Hi,

I think retention is not working as it should. I’m running many repos but today (after two years) I’m wanted to change something in overall backup process. This repo is around 500GB. But I have also larger, 4TB od 6TB. Retention is very important and retention --keep-within-daily 7d keep too many old snapshots. For example snapshot from 2023-12-07 (and many others) was left untouched with reason=daily within 7d. Why? Today is 2024-09-22

Restic version 0.17.1 on macos (homebrew)

Thanks

2

When you created your post, you were asked to provide a set of information such as the restic version, commands, all of their output, etc. You did not do this, and for that reason it is impossible for us to answer exactly why you are seeing what you are seeing. I can tell you that retention is working as it should, but unless you provide the information you were asked to, I cannot give you the exact reason for your specific data or set of snapshots.

1 Like
3

ok, sorry

version: 0.17.1
system: macos
command: restic forget --dry-run --keep-within-daily 7d (repo, password etc is set by env variables)

RESTIC_REPOSITORY=“sftp:login@host:/path/to/repo”
RESTIC_PASSWORD=“password”

4

Christ.

Paste the complete output of that command please.

5

I’m trying. But I need to remove all fragile informations. Sorry. If you know how to get output without paths, it will be helpful.

6

There is no point in removing parts of the output, as we need the full picture to assess the issue. In particular, the paths which you say you want to remove, are most likely the very key to knowing exactly what is causing the issue.

So paste it all without removing anything. If there’s something that is really sensitive, replace specific words with another word, but do so consistently so that you don’t for example replace “foo” and “bar” with “star”, because then when we are seeing “star” we will not know that this word was actually two separate words.

The best thing is if you just leave the output as it is.

7

If it is problematic for you, you can skip ahead to the likely solution which is the --group-by option to the forget command. Using that, you can control what metadata restic uses to group the snapshots before forgetting them per group according to the policy you specified. See restic help forget or Removing backup snapshots — restic 0.17.1 documentation for more information about this option.

1 Like
8

Long output. I needed to use pastebin, because of post limitations.

These “/path/to/files” are normal local path to folders on computer drive. Nothing fancy.

https://pastebin.com/raw/rLQ8faPP

9

The output you showed contains exactly what I cautioned you not to do - there are several occurrences of “/path/to/files” that each has replaced different paths instead of just one single path. This makes impossible to know which paths were different from others, and for that reason the output is not useful to debug your problem.

Since you are unable to provide the information we need to tell you the exact specific reason for your problem, I can only refer you to the --group-by option of the forget command that I mentioned in my previous post. It is most likely the solution to your problem.

1 Like
10

So there is no way to send you the result without showing the paths. Grouping will not solve this problem. These “/path/to/files” are normal local paths to folders on computer drive, but very sensitive unfortunately. Nothing fancy.

Still, as you can see retention uses only one parameter and there are many snapshots from last year.

11

It is not a question of these paths being fancy or not. CONSISTENTLY replacing"sensitive-one" with “foo” and “sensitive-two” with “bar” etc should be ok.

1 Like
12

I think it will, but I can’t prove it when you don’t provide the needed information to debug your problem. You could always try it, e.g. set --group-by host,tags and see how it makes the forget result different. You can also add --dry-run to see what it would be without actually performing the forgetting of snapshots.

There is. You just have to censor the information the right way, as explained earlier. Then your pastebin would be useful.

1 Like
13

ok, this should be fine. Sorry, for removing “----” lines, because I had to optimize weight (stupid 512kb limit)

https://pastebin.com/raw/QrCTwJUn

14

Thanks, that looks a lot better! Can you now give us a few examples of snapshot IDs you did not expect to see kept?

15

You have a lot of snapshots and a lot of different paths - it is not easy to stay on top of such setup IMO and easy to make mistakes (as by default --group-by host,paths is used for forget). I would tag all snapshots based on my intended grouping and then:

restic forget [forget details] --group-by tags

This way tags will create separate snapshots groups your forget policy is applied individually to each group.

16

Sorry for late response, but I’m new user and get limited with number of posts.

I think I need to better understand how retention works, because for me everything look wrong in this example. If you choose timeframe “to keep” everything else should be removed. Maybe I’m wrong in understanding this.

For example using this from restic docs it also does not work as in description. That was the reason I assume retention in restic is not working as it should.

https://restic.readthedocs.io/en/stable/060_forget.html#removing-snapshots-according-to-a-policy

You might want to maintain the same policy as in the example above, but have irregular backups. For example, the 7 snapshots specified with --keep-daily 7 might be spread over a longer period. If what you want is to keep daily snapshots for the last week, weekly for the last month, monthly for the last year and yearly for the last 75 years, you can instead specify forget --keep-within-daily 7d --keep-within-weekly 1m --keep-within-monthly 1y --keep-within-yearly 75y (note that 1w is not a recognized duration, so you will have to specify 7d instead).

17

Sorry to hear that :confused: You are indeed welcome, even if the forum software doesn’t seem to think so :slight_smile:

Can you tell us a few of the snapshot IDs in your paste that you did not expect to see kept?

18

Yes, of course. From my point of view retention is not working as it should because when I set time frame 7d parameter is should consider (if not set differently with other parameter) all snapshots and left only those from 7 days back.

In documentation there is an example (see below) and it is not working like is should in my repo. Why?

You might want to maintain the same policy as in the example above, but have irregular backups. For example, the 7 snapshots specified with --keep-daily 7 might be spread over a longer period. If what you want is to keep daily snapshots for the last week, weekly for the last month, monthly for the last year and yearly for the last 75 years, you can instead specify forget --keep-within-daily 7d --keep-within-weekly 1m --keep-within-monthly 1y --keep-within-yearly 75y (note that 1w is not a recognized duration, so you will have to specify 7d instead).

So answering to you question all snapshots older then 7 days should be deleted. For sure those from 2023.

I’m not saying that I’m right, maybe I don’t understand something here.

Thank you

19

You still did not provide me with what I asked for several times, actual snapshot IDs. This is annoying and not constructive.

Let me ask you a question: Imagine that you run a backup of your computer every day for five days. Then you go on vacation for two weeks and during this time you don’t make any backups at all. When you get back, your computer runs the forget command to clean out old snapshots, and it does so specifying --keep-within-daily 7d.

Would you then want it to delete all those five snapshots you had, because they are older than the seven days, or would you rather see that those five snapshots are kept as they are within seven days of the most recent snapshot so that you don’t end up with no snapshots at all in your backup?

I think this is what you are seeing. Restic is relating the seven days you specify in the forget command to the most recent snapshot, rather than the date when you run the forget command :slight_smile:

Of course it also groups the snapshots per host,path currently, so you might still want to look into using --group-by as well.

1 Like
20

Sorry, because I would have to copy all IDs older than 7d. For me the date is the main parameter.

So yes, if parameter “–keep-last 10” or “–keep-last 30” is not set then yes. It should inform that it will delete all snapshots and wait for confirmation (or not wait or come parameter --confirm-all-questions yes or --ignore-all-questions yes). This may sound stupid, but restic should do what the user tells it to do. Stupid user, sad effect o_O Maybe some “recovery” option should be avaliable until --prune is used?

OR

There should be some default retention (if not set differently by use) for example
–keep-within-daily 7d --keep-within-weekly 1m --keep-within-monthly 1y --keep-within-yearly 5y

I’m just want to say that right not, restic is doing great job but no one understand why. Can you explain why for example snapshot IDs 08b4ef33, 8aba2d2f, 85e91a83 were not set for deletion? Because it is in 7d range in that December week, 9 months ago?

Or can you explain why when I use example from docs restic is leaving me snapshots that it shouldn’t?

Whatever policy I choose, restic acts differently than I expected.

Again, I just don’t understand what the developer intended when creating the retention policy.

If restic is working as you guys indented to work, then it’s fine. Maybe some day I will understand it’s policy.