Our plan is to (eventually) release a new version with compression support, which will then be available via restic self-update.
By default, restic will always keep repositories at the version they have been initialized with, because upgrading will make it incompatible with older versions of restic that do not support the new format yet. There will be a restic migrate command though which can be used to upgrade a repo, but that will always be a manual, explicit action.
At some point we will make v2 repositories (with compression) the default for newly initialized repos.
I’ve been following the spirited and detailed discussions on the issue thread and would like to thank all of you for your thoughtful consideration of how best to implement this, it fosters a strong sense of trust to see software developed using the “move slow and make things work” mantra. @MichaelEischer you appear to have been working away ceaselessly and your hard work is appreciated. (Currently restoring some customer data saved by restic, as I type this).
To try out the compression PR, there’s also the option to copy an existing repository:
First create a new one using restic init --copy-chunker-params --repository-version 2 --repo2 ... and then use copy to transfer snapshots from the old to the new repository. The copy command has also received a few performance improvements and should be able to copy snapshots as fast as your network connection allows it.
So this is pretty neat! I copied my ~/Library/Logs folder out, backed up with restic_v1, then restic_v2 using --compress off / auto / max. I then also compressed the folder with a few standard archivers to compare with the restic repositories. About the only thing that beats restic is 7-zip -mx=9, lrzip -L 9, xz -9, zstd --22 --ultra --long, and zpaq -m5 - so just the highest levels of LZMA2, Zstandard, and ZPAQ. And they all took waayyyy longer than restic did!
wow, that really lookgs great. OK, logs are very good compressable gut its great to see those numbers. i suppose its just the initial backup. what about a few snapshots. how will they affect repo size?
Awesome! looks like I picked the right time to get a recommendation from a friend for a cool backup software that I could use for my NAS
Are there other super experimental changes on master right now? If not I might want to set up a restic cloud backup with this new version to have the repository in the new format with compression right away. (if that goes bad I could still recover from a local backup or the current cloud backup setup)
testing on my M1 mac has gone well so far!
a small test backup with max compression was 845MB vs 928MB auto vs 935MB off. Source Folder was 936MB. I probably picked the worst possible example though since more than 800MB of that was randomly generated binary data… forgot about that one…
First time using the new build on my iMac to back up a Window’s user profile after running ddrescue on a dying disk, mounting via Dislocker, and running Restic to grab the good stuff!
Total data backed up: 272.32 GiB
Total data after dedup: 179.08 GiB - 66% original size
Total data actually written to disk (compressed): 118.56 GiB - 44% original size!
That was at auto. I intend to mostly back up using max compression, but in this case I was on a time crunch! I’m currently copying my old repository over at max compression, and it’s took all week and is a little over halfway done haha
With that in mind, it copied everything in 1:15:57. So, it read at roughly 59.72 MB/s! This was from a M.2 SATA SSD to a Fusion Drive repo (1TB SSD cache + 5TB HDD). Normally I get 300MB/s easily out of it when the cache isn’t full, but I had been copying the old repo over so it had probably slowed down to HDD speeds.