I’ve been following the spirited and detailed discussions on the issue thread and would like to thank all of you for your thoughtful consideration of how best to implement this, it fosters a strong sense of trust to see software developed using the “move slow and make things work” mantra. @MichaelEischer you appear to have been working away ceaselessly and your hard work is appreciated. (Currently restoring some customer data saved by restic, as I type this).
7 Likes
Impressed to read these news! Thank you so much for all that work and this wonderful piece of software.
2 Likes
To try out the compression PR, there’s also the option to copy an existing repository:
First create a new one using restic init --copy-chunker-params --repository-version 2 --repo2 ... and then use copy to transfer snapshots from the old to the new repository. The copy command has also received a few performance improvements and should be able to copy snapshots as fast as your network connection allows it.
8 Likes
So this is pretty neat! I copied my ~/Library/Logs folder out, backed up with restic_v1, then restic_v2 using --compress off / auto / max. I then also compressed the folder with a few standard archivers to compare with the restic repositories. About the only thing that beats restic is 7-zip -mx=9, lrzip -L 9, xz -9, zstd --22 --ultra --long, and zpaq -m5 - so just the highest levels of LZMA2, Zstandard, and ZPAQ. And they all took waayyyy longer than restic did!
9 Likes
Wow. Looks like some savings at storage providers are coming up 
3 Likes
wow, that really lookgs great. OK, logs are very good compressable gut its great to see those numbers. i suppose its just the initial backup. what about a few snapshots. how will they affect repo size?
Markus
Awesome! looks like I picked the right time to get a recommendation from a friend for a cool backup software that I could use for my NAS 
Are there other super experimental changes on master right now? If not I might want to set up a restic cloud backup with this new version to have the repository in the new format with compression right away. (if that goes bad I could still recover from a local backup or the current cloud backup setup)
testing on my M1 mac has gone well so far!
a small test backup with max compression was 845MB vs 928MB auto vs 935MB off. Source Folder was 936MB. I probably picked the worst possible example though since more than 800MB of that was randomly generated binary data… forgot about that one…
2 Likes
@ Restic Team: Thank you very much, I look forward to my tests with the compression.
1 Like
Just checking: performing a restic copy from a repo at v1 to a repo at v2 will compress everything?
Yes, new data added to a v2 repo always get compressed.
1 Like
This will not work with rclone repos, I guess?
And I think restic copy --compression max will copy using max compression?
The compression support works independently from the storage backend.
It will copy all data that does not yet exist in the destination repository using maximum compression.
1 Like
Would it make sense to add a flag like --cignore "*.zip" to skip compression for .zip files, for example?
1 Like
First time using the new build on my iMac to back up a Window’s user profile after running ddrescue on a dying disk, mounting via Dislocker, and running Restic to grab the good stuff!
Total data backed up: 272.32 GiB
Total data after dedup: 179.08 GiB - 66% original size
Total data actually written to disk (compressed): 118.56 GiB - 44% original size!
3 Likes
Cool! This is at max compression?
That was at auto. I intend to mostly back up using max compression, but in this case I was on a time crunch! I’m currently copying my old repository over at max compression, and it’s took all week and is a little over halfway done haha
With that in mind, it copied everything in 1:15:57. So, it read at roughly 59.72 MB/s! This was from a M.2 SATA SSD to a Fusion Drive repo (1TB SSD cache + 5TB HDD). Normally I get 300MB/s easily out of it when the cache isn’t full, but I had been copying the old repo over so it had probably slowed down to HDD speeds.
Awesome - this is fantastic!
Can someone share some information about the type of compression algorithm used and why that was chosen?
I tried to find it in the documentation, but didn’t have any luck.
The statistics already posted show that it seems to be efficient and performant, so that is great to hear.