Cron based script creates abnormal amount of snapshots from single command

Hello all, I’ve got a Cron job to backup my system to a restic repo.

restic version:
restic 0.16.0 compiled with go1.20.6 on linux/amd64

The script, snapshots and logs are below.

The script looks as follows:

#!/bin/bash
set -euo pipefail
IFS=$'\n\t'

backup_path=/data/backup/restic

export RESTIC_PASSWORD_FILE=/root/.restic_passwd

now="$(date)"
echo "================================================="
echo "Current date and time $now"
echo "================================================="

if [[ ! $(whoami) =~ "root" ]]; then
exit
fi

restic -r $backup_path backup --verbose /etc /home /usr /root /var /docker -exclude='var/tmp' --exclude='var/lock' --exclude='var/run' --exclude='var/lib/docker' --exclude='docker/duplicacy/cache' --exclude='docker/plex/Cache'  --exclude='/docker/photoprism/cache' --exclude='/docker/photoprism/sidecar' --exclude='/docker/mariadb/data'

restic -r $backup_path prune
restic -r $backup_path check

This cronjob is scheduled for 4am with:

* 4 * * * /root/serverbackup.sh | tee -a /var/log/restic

But the snapshots look as follows:


f00a896a  2023-09-01 04:00:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

ddf061c7  2023-09-01 04:02:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

bc55f73b  2023-09-01 04:03:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

8861bddd  2023-09-01 04:04:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

5f472f24  2023-09-01 04:05:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

8c9ae5b7  2023-09-01 04:06:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

c6ece5c0  2023-09-01 04:07:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

d735c0bb  2023-09-01 04:08:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

a4441e00  2023-09-01 04:09:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

5e5d656d  2023-09-01 04:10:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

1acfb563  2023-09-01 04:11:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

367e8a8e  2023-09-01 04:12:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

2fe982b1  2023-09-01 04:13:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

036a7688  2023-09-01 04:14:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

b84ac7ac  2023-09-01 04:15:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

c4d4a959  2023-09-01 04:16:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

20f28dfd  2023-09-01 04:17:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

6085e9e8  2023-09-01 04:18:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

deba29ca  2023-09-01 04:19:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

24046645  2023-09-01 04:20:00  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

2821886e  2023-09-01 04:22:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

167bee82  2023-09-01 04:23:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

2e5e7a28  2023-09-01 04:24:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

6214c19a  2023-09-01 04:26:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

e57690b4  2023-09-01 04:27:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

af0522fc  2023-09-01 04:28:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

7241de1d  2023-09-01 04:29:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

80f919fd  2023-09-01 04:30:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

83c336bd  2023-09-01 04:31:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

5708eac8  2023-09-01 04:32:02  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

5fa4ab1e  2023-09-01 04:33:02  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

ba8bd71d  2023-09-01 04:34:00  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

2d4d23af  2023-09-01 04:35:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

05d62af6  2023-09-01 04:36:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

122cb6a7  2023-09-01 04:37:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

dfd96e60  2023-09-01 04:38:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

2ff46d80  2023-09-01 04:39:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

403b5d7b  2023-09-01 04:40:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

574df81c  2023-09-01 04:41:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

ac112485  2023-09-01 04:42:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

2e9edf87  2023-09-01 04:43:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

55b4674d  2023-09-01 04:44:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

642124fb  2023-09-01 04:45:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

6cab1009  2023-09-01 04:46:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

9c79c169  2023-09-01 04:47:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

3a05f2b5  2023-09-01 04:48:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

24f23d9a  2023-09-01 04:49:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

df8d4fa8  2023-09-01 04:50:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

65aba2c8  2023-09-01 04:51:02  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

5d46ebc9  2023-09-01 04:52:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

50b78085  2023-09-01 04:53:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

c8431268  2023-09-01 04:54:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

702402fe  2023-09-01 04:55:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

33377dd3  2023-09-01 04:56:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

c57e4314  2023-09-01 04:57:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

3207e34b  2023-09-01 04:58:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

f6703e70  2023-09-01 04:59:01  dolores                 /docker
                                                       /etc
                                                       /home
                                                       /root
                                                       /usr
                                                       /var

own log at /var/log/restic

=================================================
Current date and time Fri  1 Sep 04:00:01 UTC 2023
=================================================
open repository
lock repository
using parent snapshot bde18a73
load index files
start scan on [/etc /home /usr /root /var /docker]
start backup on [/etc /home /usr /root /var /docker]
scan finished in 19.312s: 201940 files, 15.552 GiB

Files:          81 new,   790 changed, 201069 unmodified
Dirs:           21 new,   206 changed, 67791 unmodified
Data Blobs:    709 new
Tree Blobs:    219 new
Added to the repository: 342.695 MiB (342.757 MiB stored)

processed 201940 files, 15.552 GiB in 0:38
snapshot f00a896a saved
loading indexes...
loading all snapshots...
finding data that is still in use for 323 snapshots
[0:11] 100.00%  323 / 323 snapshots

searching used packs...
collecting packs for deletion and repacking
[0:04] 100.00%  1814 / 1814 packs processed


to repack:            72 blobs / 2.328 MiB
this removes:          0 blobs / 0 B
to delete:             0 blobs / 0 B
total prune:           0 blobs / 0 B
remaining:        306013 blobs / 25.131 GiB
unused size after prune: 993.269 MiB (3.86% of remaining size)

repacking packs
[0:00] 100.00%  72 / 72 packs repacked

rebuilding index
=================================================
Current date and time Fri  1 Sep 04:01:01 UTC 2023
=================================================
open repository
lock repository
repo already locked, waiting up to 0s for the lock
[0:02] 100.00%  1744 / 1744 packs processed

deleting obsolete index files
[0:00] 100.00%  63 / 63 files deleted

removing 72 old packs
[0:00] 100.00%  72 / 72 files deleted

done
using temporary cache in /tmp/restic-check-cache-3370518544
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:25] 100.00%  323 / 323 snapshots

no errors were found
=================================================
Current date and time Fri  1 Sep 04:02:01 UTC 2023
=================================================
open repository
lock repository
using parent snapshot f00a896a
load index files
start scan on [/etc /home /usr /root /var /docker]
start backup on [/etc /home /usr /root /var /docker]
scan finished in 6.050s: 201940 files, 15.552 GiB

Files:           0 new,    15 changed, 201925 unmodified
Dirs:            0 new,    26 changed, 67992 unmodified
Data Blobs:     39 new
Tree Blobs:     27 new
Added to the repository: 24.747 MiB (24.751 MiB stored)

processed 201940 files, 15.552 GiB in 0:27
snapshot ddf061c7 saved
loading indexes...
loading all snapshots...
finding data that is still in use for 324 snapshots
[0:11] 100.00%  324 / 324 snapshots

searching used packs...
collecting packs for deletion and repacking
[0:00] 100.00%  1747 / 1747 packs processed


to repack:             0 blobs / 0 B
this removes:          0 blobs / 0 B
to delete:             0 blobs / 0 B
total prune:           0 blobs / 0 B
remaining:        306079 blobs / 25.156 GiB
unused size after prune: 993.269 MiB (3.86% of remaining size)

done
using temporary cache in /tmp/restic-check-cache-3475269345
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:10] 100.00%  324 / 324 snapshots

no errors were found
=================================================
Current date and time Fri  1 Sep 04:03:01 UTC 2023
=================================================
open repository
lock repository
using parent snapshot ddf061c7
load index files
start scan on [/etc /home /usr /root /var /docker]
start backup on [/etc /home /usr /root /var /docker]
scan finished in 4.773s: 201940 files, 15.552 GiB

Files:           0 new,    14 changed, 201926 unmodified
Dirs:            0 new,    25 changed, 67993 unmodified
Data Blobs:     39 new
Tree Blobs:     26 new
Added to the repository: 21.572 MiB (21.576 MiB stored)

processed 201940 files, 15.552 GiB in 0:25
snapshot bc55f73b saved
loading indexes...
loading all snapshots...
finding data that is still in use for 325 snapshots
[0:13] 100.00%  325 / 325 snapshots

searching used packs...
collecting packs for deletion and repacking
[0:00] 100.00%  1750 / 1750 packs processed


to repack:             0 blobs / 0 B
this removes:          0 blobs / 0 B
to delete:             0 blobs / 0 B
total prune:           0 blobs / 0 B
remaining:        306144 blobs / 25.177 GiB
unused size after prune: 993.269 MiB (3.85% of remaining size)

done
using temporary cache in /tmp/restic-check-cache-3620745517
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:11] 100.00%  325 / 325 snapshots

no errors were found

etc etc etc

It is not scheduled to run at 04:00 only

but also at 04:01, 04:02…up to 04:59 - 60 times in total every day.

Your logs and snapshots list show that everything works as configured.

Please do your reading on how cron schedule work:)

Also this is not restic issue at all - I suggest your OS forums for further inquiries how things work.

1 Like

@ndewijer Kudos for writing a good problem description, including relevant info such as restic version, the entire script used, the output showing the symptoms, etc!

To see the documentation for crontab, use man 5 crontab :+1:

True - most people do not do this. This ticket is perfect - if only related to restic…

and one more tip.

you can use:

to test your crontab schedule

1 Like

I can’t believe I missed that. Been hitting my head against this wall for a few days… I even HAVE a few entries in crontab that have it set correctly.

I apologize. I should be better than this.

2 Likes

no worries. It is always easier to spot somebody else mistake than our own.

1 Like