When does deduplication and or compression start? I couldn't see happening in the first backups

Getting Help
1

I didn’t see Restic 16.5 (default settings) do any compression or deduplication when a new empty repository receives a backup for the first time. Tested this a couple of times and got the same result. Is this normal and does this mean if you gave it a larger amount of data to backup it would produce the same result or do files have to be of certain size before any saving occurs?

  • Test 1 : Source data: 166 GiB, 34,000 files, each file 5 MiB of random ASCI chars. Compressed by pigz gave 16.9% saving, restic gave 0% savings on the same data.
  • Test 2 : Source data:190 GiB, 78,000 files with sizes ranging from 1 KiB to 5 MiB filled with random ASCI chars. Compressed by pigz to gave 16.3% savings, restic gave 0.01% savings on the same data.

Normally with inline data deduplication/compression I would have expected some savings when the source data is compressible. Some deduplication/compression tech advertises when deduplication starts, I couldn’t find it any in of the restic doco.

My guess is there maybe a point when a file is too small to be considered for deduplication - I’ve seen that in some deduplication apps but no mention of it the restic doco.

I did see some very efficient removal of identical files but that was not in an empty repository.

2

THIS is why there is no deduplication. Full randomness practically guarantees no repetition of identical blocks of a significant size.

3

Agree, if input is completely random that is true and that is why deduplication doesn’t like source data that is encrypted, compressed* or generated by /dev/random or /dev/urandom. Restricting randomness to the range of 95 printable ASCI characters should have reduced randomness. I’ll test again with files filled with random data based on restricted sets of 60, 26 chars. Did quick with 10chars (0-9) got about 50% savings.

My guess compression in restic is like other deduplication apps, after the input has been chopped into chunks for deduplication each unique chunk is individually compressed before its stored in the repository. Interesting that pigz found something to compress, not restic.

*The only compressed data that I’m aware of that is deduplication friendly is data compressed by gzip or pigz using the --rsyncable option.

4

At this stage I don’t understand what the purpose of this thread is anymore. Deduplication in restic works. It is that simple. :slight_smile:

If you experience problems, don’t hesitate to ask.

2 Likes
5

And here few more details how it works in restic:

https://restic.net/blog/2015-09-12/restic-foundation1-cdc/

1 Like
6

Testing stuff you get to know how it works and its limits for storing stuff - storage is where the cost is.

7

Thanks, looks like what others call “variable chunks” or “variable blocks”