Strange behavior during the second backup

Getting Help
1

Before I open an issue on github I wanted to ask here if I have misunderstood something or am doing something fundamentally wrong.

I made a backup and sent the data and to the restic rest server. So far so good.

Files:       8543163 new,     0 changed,     0 unmodified
Dirs:        1187635 new,     0 changed,     0 unmodified
Data Blobs:  4282180 new
Tree Blobs:  1116744 new
Added to the repo: 1020.552 GiB

processed 8543163 files, 1.423 TiB in 8:36:53
snapshot 56720fd1 saved

real    516m54,233s
user    463m21,540s
sys     71m58,028s

Today I created a second snapshot and here is something I don’t understand.

Files:       8545104 new,     0 changed,     0 unmodified
Dirs:        1188092 new,     0 changed,     0 unmodified
Data Blobs:   6794 new
Tree Blobs:   8672 new
Added to the repo: 5.098 GiB

processed 8545104 files, 1.425 TiB in 5:05:45
snapshot 40604ea1 saved

real	305m45,765s
user	254m3,288s
sys	27m18,796s

What surprises me is why the number of new files is so large during the second run. I have hardly changed anything in my files in the meantime. I would have expected restic to recognize this.

The command i run in both cases:
restic backup /home/usr*/home/www/ /home/usr*p* /usr/src/nextco/ --exclude-file /root/exclude -v

The Source FS is a is an ext4 on an md device what hdds uses. And the ext4 is mounted with rw,noatime,quota,usrquota,stripe=256,data=ordered

btw. Thank you for this great software!

2

This probably has to do with metadata and filesystem factors. The amount of data added is probably because restic backed up the metadata for all those 8.5 million files.

Please edit your initial post so that it contains the complete commands you used to run the backups, and please also let us know what the source storage/disk/filesystem/mount is. Thanks!

1 Like
3

Did maybe the hostname change between backup runs? Are the * in the backup command just for obfuscating parts of the filename or are those real * which would be expanded by a shell. In the latter case: Did that argument list expand to the same paths both times?

1 Like
4

@l1f It would be nice if you could also paste the output of the snapshots restic command :slight_smile:

5

The hostname did not change between the two runs.

Exactly, the * are real and are expanded in the shell and that also in both cases the same. (Although I can not say that with absolute confidence in retrospect, in home are very many folders).

6

@rawtaz The output of the command is a bit longer ^^ So about 1600 lines long, do you still want to see the output?

7

ATM, restic is only able to detect parent snapshots if the paths to backup are exactly identical. If you use * and they resolve to changed paths, restic will make a “new” backup, i.e. it reads and chunks all files and only does not save already existing chunks.

Another drawback is that that all paths show in the snapshots list which may be not intended.

I suggest you use a fixed path and (if needed) exclude the things you don’t want to backup. Another possibility is to wait for

to manually set a suitable path, or

which might bring you the behavior you are expecting…

1 Like
8

Ah, I see. I’ll try it that way. Thanks for the information!