the highly anticipated PR #3666 was just merged, so now there is support for compression in master! This means that the repository format changes for compression are finalized (unless we find something very critical, which I don’t expect). I’d like to especially thank @MichaelEischer, who did most of the work, and all the other people who discussed this large change with us!
For now, repository version v2 is not enabled by default, and there is no way (yet) for migrating an existing repo. That will all follow in PR #3704 a bit later.
If you want to play with it:
Build restic from master (or fetch the binary from the beta builder)
Initialize a new repo like this: restic init --repository-version 2
Compression is now available and enabled by default (for that repo). You can set the desired compression level by passing it to --compression (e.g. restic backup --compression max), supported are auto, max and off.
Please let us know how it goes and report all issues you find!
I don’t mean existing repositories are compressed automatically by the new version of restic. Rather, compression is enabled in newly created v2 repositories (and migration occurs by something like restic-migrate).
I would say when the core team feels like this new feature is ready for it. Probably after rigorous testing and potential bug fixes etc.
This is so brand new that it just doesn’t make sense to give any time line for it. If you want to use it, use the flags to enable it. There are also folks which don’t want to use it and this major change needs to be communicated over a long period of time to make sure folks know what’s what
Funny that the first post for this major announcement is “when ETA” lol
Huge congratulations to the Maintainers for landing this
Our plan is to (eventually) release a new version with compression support, which will then be available via restic self-update.
By default, restic will always keep repositories at the version they have been initialized with, because upgrading will make it incompatible with older versions of restic that do not support the new format yet. There will be a restic migrate command though which can be used to upgrade a repo, but that will always be a manual, explicit action.
At some point we will make v2 repositories (with compression) the default for newly initialized repos.
I’ve been following the spirited and detailed discussions on the issue thread and would like to thank all of you for your thoughtful consideration of how best to implement this, it fosters a strong sense of trust to see software developed using the “move slow and make things work” mantra. @MichaelEischer you appear to have been working away ceaselessly and your hard work is appreciated. (Currently restoring some customer data saved by restic, as I type this).
To try out the compression PR, there’s also the option to copy an existing repository:
First create a new one using restic init --copy-chunker-params --repository-version 2 --repo2 ... and then use copy to transfer snapshots from the old to the new repository. The copy command has also received a few performance improvements and should be able to copy snapshots as fast as your network connection allows it.
So this is pretty neat! I copied my ~/Library/Logs folder out, backed up with restic_v1, then restic_v2 using --compress off / auto / max. I then also compressed the folder with a few standard archivers to compare with the restic repositories. About the only thing that beats restic is 7-zip -mx=9, lrzip -L 9, xz -9, zstd --22 --ultra --long, and zpaq -m5 - so just the highest levels of LZMA2, Zstandard, and ZPAQ. And they all took waayyyy longer than restic did!
wow, that really lookgs great. OK, logs are very good compressable gut its great to see those numbers. i suppose its just the initial backup. what about a few snapshots. how will they affect repo size?
Awesome! looks like I picked the right time to get a recommendation from a friend for a cool backup software that I could use for my NAS
Are there other super experimental changes on master right now? If not I might want to set up a restic cloud backup with this new version to have the repository in the new format with compression right away. (if that goes bad I could still recover from a local backup or the current cloud backup setup)
testing on my M1 mac has gone well so far!
a small test backup with max compression was 845MB vs 928MB auto vs 935MB off. Source Folder was 936MB. I probably picked the worst possible example though since more than 800MB of that was randomly generated binary data… forgot about that one…
This means that the repository format changes for compression are finalized (unless we find something very critical, which I don’t expect). I’d like to especially thank @MichaelEischer, who did most of the work, and all the other people who discussed this large change with us!
