Scanning file system vs. deduplicated repo

Getting Help
1

restic 0.19.1 compiled with go1.26.5 on darwin/arm64: workstation
restic 0.19.1 compiled with go1.26.5 on darwin/amd64: laptop

I would like to make sure I have a correct understanding of scanning the filesystem for changes and deduplication of files in the repository.

I will be taking a laptop on a two-week trip. Every day of the trip, I will be shooting many digital photographs and copying them to the laptop. From the laptop, restic will back up every day’s photos to an existing repository at Backblaze.

After the trip is over and I return home, I will rsync all the photos from my laptop to my main desktop workstation. The image files will be placed into a directory structure on the workstation where all my digital images, taken over 20+ years, are stored.

Restic backs up the workstation to the same repository that was used by the laptop. Of course, both the host and the path metadata are different between laptop and workstation.

Do I understand correctly that since the laptop already backed up the images from the trip, deduplication will ensure that the same images will not take up new space in the repository when the workstation is backed up? I do not care if the workstation must rescan the files to notice they are unchanged, but I don’t want the files to be re-uploaded.

The image files are large (50MB+ each); this is why I want to make sure I understand deduplication correctly.

Thank you.

P.S./Edit/Clarification: The image files are individual files on the laptop and the workstation. They will not be incorporated into an assets database or something similar where their identity as files will be lost; however, a file named “a” on the laptop may be named “x” on the workstation with identical contents.

2

Hi @thomask

The actual data of the image files will be deduplicated - if you are using the identical chunker parameters on the laptop and the workstation.

However, metadata (like file names, user information etc.) may not be fully deduplicated. It depends on the rsync mechanism and file systems used whether tree objects can or cannot be deduplicated. In any case you’ll get a new snapshot.

However, for digital images metadata is only a small fraction of the actual data, so I expect only some kiB’s to be uploaded.

3

Thanks, @alexweiss. How can I make sure I’m using the same chunker parameters? I thought, from reading the documentation, that those parameters are set in stone when the repository is created…in this case, both machines are using the same repository. Is there something more I should do to ensure the same parameters? I use the same pack size of 96M on both machines, if that makes any difference.

4

Then nothing to worry about. I think @alexweiss misread a bit your original post.

Chunker parameter only matters when copying data between different repositories.

5

Ah, sorry for the confusion about chunking parameters. Of course, if this is the same repository, the same chunking parameters are used…