Retention: Delete data when not found for 7 days

Getting Help
1

Hello!

I have compiled restic in debug mode and try to fiddle around with a test repo.
Imagine a folder structure like this

└── folder1
    ├── testfile
    └── testfile2

2 directories, 2 files

with eventually more folders next to “folder1”.
I’ll create backups of the parent folder of “folder1” for example everyday for 14 days.

#!/bin/bash

for i in {0..14}; do
    timestamp=$(gdate +"%Y-%m-%d 12:00:00" -d "+ ${i} days")
    echo "=========================================================="
    echo "run backup on ${timestamp}"
    ./restic backup --quiet --time "${timestamp}" $HOME/tmp/restic_testdir
    ./restic forget --compact --keep-daily 7
    ./restic prune
done

./restic snapshots

./restic snapshots
repository cddc6c6e opened (version 2, compression level auto)
ID        Time                 Host                     Tags        Paths
-------------------------------------------------------------------------------------------------------
4010a517  2024-03-31 12:00:00  MacBook-Pro-2.fritz.box              /Users/p.querner/tmp/restic_testdir
d5b79ac1  2024-04-01 12:00:00  MacBook-Pro-2.fritz.box              /Users/p.querner/tmp/restic_testdir
3acff65c  2024-04-02 12:00:00  MacBook-Pro-2.fritz.box              /Users/p.querner/tmp/restic_testdir
08a296e1  2024-04-03 12:00:00  MacBook-Pro-2.fritz.box              /Users/p.querner/tmp/restic_testdir
4540aa84  2024-04-04 12:00:00  MacBook-Pro-2.fritz.box              /Users/p.querner/tmp/restic_testdir
35638cac  2024-04-05 12:00:00  MacBook-Pro-2.fritz.box              /Users/p.querner/tmp/restic_testdir
e1421443  2024-04-06 12:00:00  MacBook-Pro-2.fritz.box              /Users/p.querner/tmp/restic_testdir
-------------------------------------------------------------------------------------------------------
7 snapshots
./restic snapshots --json | jq -r '.[].id' | xargs -I {} ./restic ls --long {}
repository cddc6c6e opened (version 2, compression level auto)
[0:00] 100.00%  1 / 1 index files loaded
snapshot 4010a517 of [/Users/p.querner/tmp/restic_testdir] at 2024-03-31 12:00:00 +0200 CEST by p.querner@MacBook-Pro-2.fritz.box filtered by []:
drwxr-xr-x     0    80      0 2024-01-20 13:18:56 /Users
drwxr-x---   502    20      0 2024-03-23 19:12:08 /Users/p.querner
drwxr-xr-x   502    20      0 2024-03-23 17:49:55 /Users/p.querner/tmp
drwxr-xr-x   502    20      0 2024-03-23 19:03:18 /Users/p.querner/tmp/restic_testdir
drwxr-xr-x   502    20      0 2024-03-23 19:03:28 /Users/p.querner/tmp/restic_testdir/folder1
-rw-r--r--   502    20      9 2024-03-23 19:03:26 /Users/p.querner/tmp/restic_testdir/folder1/testfile
-rw-r--r--   502    20     10 2024-03-23 19:03:28 /Users/p.querner/tmp/restic_testdir/folder1/testfile2
repository cddc6c6e opened (version 2, compression level auto)
[0:00] 100.00%  1 / 1 index files loaded
snapshot d5b79ac1 of [/Users/p.querner/tmp/restic_testdir] at 2024-04-01 12:00:00 +0200 CEST by p.querner@MacBook-Pro-2.fritz.box filtered by []:
drwxr-xr-x     0    80      0 2024-01-20 13:18:56 /Users
drwxr-x---   502    20      0 2024-03-23 19:12:08 /Users/p.querner
drwxr-xr-x   502    20      0 2024-03-23 17:49:55 /Users/p.querner/tmp
drwxr-xr-x   502    20      0 2024-03-23 19:03:18 /Users/p.querner/tmp/restic_testdir
drwxr-xr-x   502    20      0 2024-03-23 19:03:28 /Users/p.querner/tmp/restic_testdir/folder1
-rw-r--r--   502    20      9 2024-03-23 19:03:26 /Users/p.querner/tmp/restic_testdir/folder1/testfile
-rw-r--r--   502    20     10 2024-03-23 19:03:28 /Users/p.querner/tmp/restic_testdir/folder1/testfile2
repository cddc6c6e opened (version 2, compression level auto)
[0:00] 100.00%  1 / 1 index files loaded
snapshot 3acff65c of [/Users/p.querner/tmp/restic_testdir] at 2024-04-02 12:00:00 +0200 CEST by p.querner@MacBook-Pro-2.fritz.box filtered by []:
drwxr-xr-x     0    80      0 2024-01-20 13:18:56 /Users
drwxr-x---   502    20      0 2024-03-23 19:12:08 /Users/p.querner
drwxr-xr-x   502    20      0 2024-03-23 17:49:55 /Users/p.querner/tmp
drwxr-xr-x   502    20      0 2024-03-23 19:03:18 /Users/p.querner/tmp/restic_testdir
drwxr-xr-x   502    20      0 2024-03-23 19:03:28 /Users/p.querner/tmp/restic_testdir/folder1
-rw-r--r--   502    20      9 2024-03-23 19:03:26 /Users/p.querner/tmp/restic_testdir/folder1/testfile
-rw-r--r--   502    20     10 2024-03-23 19:03:28 /Users/p.querner/tmp/restic_testdir/folder1/testfile2
repository cddc6c6e opened (version 2, compression level auto)
[0:00] 100.00%  1 / 1 index files loaded
snapshot 08a296e1 of [/Users/p.querner/tmp/restic_testdir] at 2024-04-03 12:00:00 +0200 CEST by p.querner@MacBook-Pro-2.fritz.box filtered by []:
drwxr-xr-x     0    80      0 2024-01-20 13:18:56 /Users
drwxr-x---   502    20      0 2024-03-23 19:12:08 /Users/p.querner
drwxr-xr-x   502    20      0 2024-03-23 17:49:55 /Users/p.querner/tmp
drwxr-xr-x   502    20      0 2024-03-23 19:03:18 /Users/p.querner/tmp/restic_testdir
drwxr-xr-x   502    20      0 2024-03-23 19:03:28 /Users/p.querner/tmp/restic_testdir/folder1
-rw-r--r--   502    20      9 2024-03-23 19:03:26 /Users/p.querner/tmp/restic_testdir/folder1/testfile
-rw-r--r--   502    20     10 2024-03-23 19:03:28 /Users/p.querner/tmp/restic_testdir/folder1/testfile2
repository cddc6c6e opened (version 2, compression level auto)
[0:00] 100.00%  1 / 1 index files loaded
snapshot 4540aa84 of [/Users/p.querner/tmp/restic_testdir] at 2024-04-04 12:00:00 +0200 CEST by p.querner@MacBook-Pro-2.fritz.box filtered by []:
drwxr-xr-x     0    80      0 2024-01-20 13:18:56 /Users
drwxr-x---   502    20      0 2024-03-23 19:12:08 /Users/p.querner
drwxr-xr-x   502    20      0 2024-03-23 17:49:55 /Users/p.querner/tmp
drwxr-xr-x   502    20      0 2024-03-23 19:03:18 /Users/p.querner/tmp/restic_testdir
drwxr-xr-x   502    20      0 2024-03-23 19:03:28 /Users/p.querner/tmp/restic_testdir/folder1
-rw-r--r--   502    20      9 2024-03-23 19:03:26 /Users/p.querner/tmp/restic_testdir/folder1/testfile
-rw-r--r--   502    20     10 2024-03-23 19:03:28 /Users/p.querner/tmp/restic_testdir/folder1/testfile2
repository cddc6c6e opened (version 2, compression level auto)
[0:00] 100.00%  1 / 1 index files loaded
snapshot 35638cac of [/Users/p.querner/tmp/restic_testdir] at 2024-04-05 12:00:00 +0200 CEST by p.querner@MacBook-Pro-2.fritz.box filtered by []:
drwxr-xr-x     0    80      0 2024-01-20 13:18:56 /Users
drwxr-x---   502    20      0 2024-03-23 19:12:08 /Users/p.querner
drwxr-xr-x   502    20      0 2024-03-23 17:49:55 /Users/p.querner/tmp
drwxr-xr-x   502    20      0 2024-03-23 19:03:18 /Users/p.querner/tmp/restic_testdir
drwxr-xr-x   502    20      0 2024-03-23 19:03:28 /Users/p.querner/tmp/restic_testdir/folder1
-rw-r--r--   502    20      9 2024-03-23 19:03:26 /Users/p.querner/tmp/restic_testdir/folder1/testfile
-rw-r--r--   502    20     10 2024-03-23 19:03:28 /Users/p.querner/tmp/restic_testdir/folder1/testfile2
repository cddc6c6e opened (version 2, compression level auto)
[0:00] 100.00%  1 / 1 index files loaded
snapshot e1421443 of [/Users/p.querner/tmp/restic_testdir] at 2024-04-06 12:00:00 +0200 CEST by p.querner@MacBook-Pro-2.fritz.box filtered by []:
drwxr-xr-x     0    80      0 2024-01-20 13:18:56 /Users
drwxr-x---   502    20      0 2024-03-23 19:12:08 /Users/p.querner
drwxr-xr-x   502    20      0 2024-03-23 17:49:55 /Users/p.querner/tmp
drwxr-xr-x   502    20      0 2024-03-23 19:03:18 /Users/p.querner/tmp/restic_testdir
drwxr-xr-x   502    20      0 2024-03-23 19:03:28 /Users/p.querner/tmp/restic_testdir/folder1
-rw-r--r--   502    20      9 2024-03-23 19:03:26 /Users/p.querner/tmp/restic_testdir/folder1/testfile
-rw-r--r--   502    20     10 2024-03-23 19:03:28 /Users/p.querner/tmp/restic_testdir/folder1/testfile2

With run, I will have 7 snapshots with the above folder intact.

Now imagine I have deleted that folder “folder1” in any of the backup days (after the first run, obviously).

tree ~/tmp/restic_testdir/

0 directories, 0 files

(folder1 is now deleted)

<some time passes, or I alter the script above to create 30 days not 14 days>

The next time that script is run, this is the new snapshot output:

./restic snapshots
repository cddc6c6e opened (version 2, compression level auto)
ID        Time                 Host                     Tags        Paths
-------------------------------------------------------------------------------------------------------
c21c6ca0  2024-04-16 12:00:00  MacBook-Pro-2.fritz.box              /Users/p.querner/tmp/restic_testdir
13f8156a  2024-04-17 12:00:00  MacBook-Pro-2.fritz.box              /Users/p.querner/tmp/restic_testdir
3ec4aa63  2024-04-18 12:00:00  MacBook-Pro-2.fritz.box              /Users/p.querner/tmp/restic_testdir
ba514a2e  2024-04-19 12:00:00  MacBook-Pro-2.fritz.box              /Users/p.querner/tmp/restic_testdir
086e3852  2024-04-20 12:00:00  MacBook-Pro-2.fritz.box              /Users/p.querner/tmp/restic_testdir
f18f92ab  2024-04-21 12:00:00  MacBook-Pro-2.fritz.box              /Users/p.querner/tmp/restic_testdir
f6f9c0d6  2024-04-22 12:00:00  MacBook-Pro-2.fritz.box              /Users/p.querner/tmp/restic_testdir
-------------------------------------------------------------------------------------------------------
7 snapshots
./restic snapshots --json | jq -r '.[].id' | xargs -I {} ./restic ls --long {}
repository cddc6c6e opened (version 2, compression level auto)
[0:00] 100.00%  1 / 1 index files loaded
snapshot c21c6ca0 of [/Users/p.querner/tmp/restic_testdir] at 2024-04-16 12:00:00 +0200 CEST by p.querner@MacBook-Pro-2.fritz.box filtered by []:
drwxr-xr-x     0    80      0 2024-01-20 13:18:56 /Users
drwxr-x---   502    20      0 2024-03-23 19:17:36 /Users/p.querner
drwxr-xr-x   502    20      0 2024-03-23 17:49:55 /Users/p.querner/tmp
drwxr-xr-x   502    20      0 2024-03-23 19:03:18 /Users/p.querner/tmp/restic_testdir
drwxr-xr-x   502    20      0 2024-03-23 19:03:28 /Users/p.querner/tmp/restic_testdir/folder1
-rw-r--r--   502    20      9 2024-03-23 19:03:26 /Users/p.querner/tmp/restic_testdir/folder1/testfile
-rw-r--r--   502    20     10 2024-03-23 19:03:28 /Users/p.querner/tmp/restic_testdir/folder1/testfile2
repository cddc6c6e opened (version 2, compression level auto)
[0:00] 100.00%  1 / 1 index files loaded
snapshot 13f8156a of [/Users/p.querner/tmp/restic_testdir] at 2024-04-17 12:00:00 +0200 CEST by p.querner@MacBook-Pro-2.fritz.box filtered by []:
drwxr-xr-x     0    80      0 2024-01-20 13:18:56 /Users
drwxr-x---   502    20      0 2024-03-23 19:17:36 /Users/p.querner
drwxr-xr-x   502    20      0 2024-03-23 17:49:55 /Users/p.querner/tmp
drwxr-xr-x   502    20      0 2024-03-23 19:03:18 /Users/p.querner/tmp/restic_testdir
drwxr-xr-x   502    20      0 2024-03-23 19:03:28 /Users/p.querner/tmp/restic_testdir/folder1
-rw-r--r--   502    20      9 2024-03-23 19:03:26 /Users/p.querner/tmp/restic_testdir/folder1/testfile
-rw-r--r--   502    20     10 2024-03-23 19:03:28 /Users/p.querner/tmp/restic_testdir/folder1/testfile2
repository cddc6c6e opened (version 2, compression level auto)
[0:00] 100.00%  1 / 1 index files loaded
snapshot 3ec4aa63 of [/Users/p.querner/tmp/restic_testdir] at 2024-04-18 12:00:00 +0200 CEST by p.querner@MacBook-Pro-2.fritz.box filtered by []:
drwxr-xr-x     0    80      0 2024-01-20 13:18:56 /Users
drwxr-x---   502    20      0 2024-03-23 19:17:36 /Users/p.querner
drwxr-xr-x   502    20      0 2024-03-23 17:49:55 /Users/p.querner/tmp
drwxr-xr-x   502    20      0 2024-03-23 19:03:18 /Users/p.querner/tmp/restic_testdir
drwxr-xr-x   502    20      0 2024-03-23 19:03:28 /Users/p.querner/tmp/restic_testdir/folder1
-rw-r--r--   502    20      9 2024-03-23 19:03:26 /Users/p.querner/tmp/restic_testdir/folder1/testfile
-rw-r--r--   502    20     10 2024-03-23 19:03:28 /Users/p.querner/tmp/restic_testdir/folder1/testfile2
repository cddc6c6e opened (version 2, compression level auto)
[0:00] 100.00%  1 / 1 index files loaded
snapshot ba514a2e of [/Users/p.querner/tmp/restic_testdir] at 2024-04-19 12:00:00 +0200 CEST by p.querner@MacBook-Pro-2.fritz.box filtered by []:
drwxr-xr-x     0    80      0 2024-01-20 13:18:56 /Users
drwxr-x---   502    20      0 2024-03-23 19:17:36 /Users/p.querner
drwxr-xr-x   502    20      0 2024-03-23 17:49:55 /Users/p.querner/tmp
drwxr-xr-x   502    20      0 2024-03-23 19:03:18 /Users/p.querner/tmp/restic_testdir
drwxr-xr-x   502    20      0 2024-03-23 19:03:28 /Users/p.querner/tmp/restic_testdir/folder1
-rw-r--r--   502    20      9 2024-03-23 19:03:26 /Users/p.querner/tmp/restic_testdir/folder1/testfile
-rw-r--r--   502    20     10 2024-03-23 19:03:28 /Users/p.querner/tmp/restic_testdir/folder1/testfile2
repository cddc6c6e opened (version 2, compression level auto)
[0:00] 100.00%  1 / 1 index files loaded
snapshot 086e3852 of [/Users/p.querner/tmp/restic_testdir] at 2024-04-20 12:00:00 +0200 CEST by p.querner@MacBook-Pro-2.fritz.box filtered by []:
drwxr-xr-x     0    80      0 2024-01-20 13:18:56 /Users
drwxr-x---   502    20      0 2024-03-23 19:17:36 /Users/p.querner
drwxr-xr-x   502    20      0 2024-03-23 17:49:55 /Users/p.querner/tmp
drwxr-xr-x   502    20      0 2024-03-23 19:03:18 /Users/p.querner/tmp/restic_testdir
drwxr-xr-x   502    20      0 2024-03-23 19:03:28 /Users/p.querner/tmp/restic_testdir/folder1
-rw-r--r--   502    20      9 2024-03-23 19:03:26 /Users/p.querner/tmp/restic_testdir/folder1/testfile
-rw-r--r--   502    20     10 2024-03-23 19:03:28 /Users/p.querner/tmp/restic_testdir/folder1/testfile2
repository cddc6c6e opened (version 2, compression level auto)
[0:00] 100.00%  1 / 1 index files loaded
snapshot f18f92ab of [/Users/p.querner/tmp/restic_testdir] at 2024-04-21 12:00:00 +0200 CEST by p.querner@MacBook-Pro-2.fritz.box filtered by []:
drwxr-xr-x     0    80      0 2024-01-20 13:18:56 /Users
drwxr-x---   502    20      0 2024-03-23 19:17:36 /Users/p.querner
drwxr-xr-x   502    20      0 2024-03-23 17:49:55 /Users/p.querner/tmp
drwxr-xr-x   502    20      0 2024-03-23 19:03:18 /Users/p.querner/tmp/restic_testdir
drwxr-xr-x   502    20      0 2024-03-23 19:03:28 /Users/p.querner/tmp/restic_testdir/folder1
-rw-r--r--   502    20      9 2024-03-23 19:03:26 /Users/p.querner/tmp/restic_testdir/folder1/testfile
-rw-r--r--   502    20     10 2024-03-23 19:03:28 /Users/p.querner/tmp/restic_testdir/folder1/testfile2
repository cddc6c6e opened (version 2, compression level auto)
[0:00] 100.00%  1 / 1 index files loaded
snapshot f6f9c0d6 of [/Users/p.querner/tmp/restic_testdir] at 2024-04-22 12:00:00 +0200 CEST by p.querner@MacBook-Pro-2.fritz.box filtered by []:
drwxr-xr-x     0    80      0 2024-01-20 13:18:56 /Users
drwxr-x---   502    20      0 2024-03-23 19:17:36 /Users/p.querner
drwxr-xr-x   502    20      0 2024-03-23 17:49:55 /Users/p.querner/tmp
drwxr-xr-x   502    20      0 2024-03-23 19:03:18 /Users/p.querner/tmp/restic_testdir
drwxr-xr-x   502    20      0 2024-03-23 19:03:28 /Users/p.querner/tmp/restic_testdir/folder1
-rw-r--r--   502    20      9 2024-03-23 19:03:26 /Users/p.querner/tmp/restic_testdir/folder1/testfile
-rw-r--r--   502    20     10 2024-03-23 19:03:28 /Users/p.querner/tmp/restic_testdir/folder1/testfile2

As you can see, the “folder1” is still available, for 7 snapshots.

So the question is: what is the parameter that only keeps the snapshot in-tact for 7 days and after that, get completely rid of it?

I tried with keep-within-daily 7 but this causes to have 1 daily snapshot for the amount of runs.

Thanks!

1 Like
2

I believe I have my answer.
restic doesnt handle that on its own, so you have to implement that with help by parsing restic snapshots --json.

This is a script which does remove a snapshot when its path could no longer be found. I’ve only tested this when the backup folders are added 1-by-1 which might have other issues when not used in a test-environment.

#!/bin/bash

function dir_exists() {
  [ -d "$1" ] && return 0 || return 1
}

SOURCE_FILE="backup_dirs.txt"

backup() {
  local dir="$1"
  restic backup "$dir"
}

forget() {
  local dir="$1"


  # Check if directory exists
  if ! dir_exists "$dir"; then
    snapshot_id_with_folder=$(restic snapshots --json | jq -r '.[] | select(.paths[] == "'"$dir"'") | .id')

    # Is path known in snapshots?
    if ! [[ -z "$snapshot_id_with_folder" ]]; then
      restic unlock
      restic forget $snapshot_id_with_folder --prune
    fi

  fi

}

BACKUP_DIRS=$(cat "$SOURCE_FILE")
for dir in $BACKUP_DIRS; do
  backup "$dir"
done

for dir in $BACKUP_DIRS; do
  forget "$dir"
done

restic forget --keep-last 1
restic prune
echo "** Post-script restic check (5% of files):"
restic check --read-data-subset=5%

backup_dirs.txt

/Users/p.querner/tmp/restic_testdir/folder1
/Users/p.querner/tmp/restic_testdir/folder2

When I remove “folder1” eventually and run this script, that folder will be completely forgotten.
The “7 days” are not solved yet, only thing comes to mind to not have daily backups but weekly.
(Or run that part weekly and everything else daily)

I kinda dont like that I iterate over the folders there, might look into it some more later.

3

If I understood your question correctly, you want to verify that a folder that was deleted on your filesystem actually disappears after 7 days from your repository.

I amended your test script to do the whole flow in 1 file.

#!/bin/bash -eux

mkdir -p ${HOME}/tmp/my-backup-dir
echo "Demo of file changes in restic" > ${HOME}/tmp/README
touch ${HOME}/tmp/my-backup-dir/hello
touch ${HOME}/tmp/my-backup-dir/world

export RESTIC_REPOSITORY=$HOME/backup
export RESTIC_PASSWORD=very-unsafe-but-works
restic init

for i in {0..7}; do
    timestamp=$(date +"%Y-%m-%d 12:00:00" -d "+ ${i} days")
    echo "=========================================================="
    echo "run backup on ${timestamp}"
    restic backup --quiet --time "${timestamp}" $HOME/tmp/
done

restic ls latest

rm -rf ${HOME}/tmp/my-backup-dir

for i in {8..14}; do
    timestamp=$(date +"%Y-%m-%d 12:00:00" -d "+ ${i} days")
    echo "=========================================================="
    echo "run backup on ${timestamp}"
    restic backup --quiet --time "${timestamp}" $HOME/tmp/
done

restic ls latest

If you run this and inspect all the snapshots, you’ll notice that the files were removed from the snapshot.
If you now prune the data

restic forget --compact --keep-daily 7 --prune

And if you now inspect all the snapshots, my-backup-dir is nowhere to be seen:

restic snapshots --json | jq -r '.[].id' | xargs -I {} restic ls --long {}