I’m looking for some clarification on the behavior I encountered when doing a snapshot.
Yesterday I set up a repo and made a snapshot of a ~660GB directory, which took almost 4 hours. I then set up a cron job that would make a snapshot of the same folder every night.
Since I haven’t touched the content of the directory, I was expecting a quick snasphot. Yet I was surprised when I checked my cron’s output this morning:
$ export RESTIC_PASSWORD_FILE=/root/restic_pwd.txt
$ restic backup -r /export/backups/photos /export/photos
Files: 68741 new, 0 changed, 0 unmodified
Dirs: 374 new, 0 changed, 0 unmodified
Added to the repo: 752 B
processed 68741 files, 665.526 GiB in 1:37:14
snapshot 5bfc49f2 saved
This raised a few questions:
Why did it marked every files as new instead of unmodified?
It seems it spent 1h37 scanning the entire folder for changes, is there a way to speed up the process?
Thanks
Edit: forgot to specify the version I’m running.
axel@helios64:~$ restic version
restic 0.10.0 compiled with go1.15.2 on linux/arm64
Can’t you run it manually (the same command of course) to see if it reproduces that way too (it should, and then you should be able to get the complete output)?
I ran the command a second time expecting it to run for more than an hour and got this:
$ export RESTIC_PASSWORD_FILE=/root/restic_pwd.txt
$ restic backup -r /export/backups/photos /export/photos
repository 23f4efef opened successfully, password is correct
Files: 0 new, 0 changed, 68741 unmodified
Dirs: 0 new, 0 changed, 374 unmodified
Added to the repo: 0 B
processed 68741 files, 665.526 GiB in 0:35
snapshot 780215dc saved
I ran the command through OpenMediaVault (I originally set up the cron with its GUI) to make sure it was not inferring with restic, got the same output as above.
Weird. I enabled the cron, I’ll see if it behaves differently tonight.
Thank you for that. Unfortunately I was missing one thing; Can you please add -v so the command is restic -v backup -r /export/backups/photos /export/photos? That way we can verify that restic uses a parent snapshot, which I presume it does.
Honestly, if the backup set you used in the cron run was the same as your initial run, then I would totally expect that restic found the parent snapshot. But if it did not, then that is one possible reason why it rescanned the files.
I don’t know what your repo is stored on but you can try using the --ignore-inode option or the --ignore-ctime option in https://github.com/restic/restic/pull/2823 (if you build your restic from that PR, as it’s not in the 0.10.0 version of restic). But it all depends on what the storage is.
Why it took so long to rescan I can’t say, I’d expect that to depend on the performance of your system’s components.