Compression support has landed in master!

I would say when the core team feels like this new feature is ready for it. Probably after rigorous testing and potential bug fixes etc.

This is so brand new that it just doesn’t make sense to give any time line for it. If you want to use it, use the flags to enable it. There are also folks which don’t want to use it and this major change needs to be communicated over a long period of time to make sure folks know what’s what :slight_smile:

Funny that the first post for this major announcement is “when ETA” lol

Huge congratulations to the Maintainers for landing this :tada:

4 Likes

Our plan is to (eventually) release a new version with compression support, which will then be available via restic self-update.

By default, restic will always keep repositories at the version they have been initialized with, because upgrading will make it incompatible with older versions of restic that do not support the new format yet. There will be a restic migrate command though which can be used to upgrade a repo, but that will always be a manual, explicit action.

At some point we will make v2 repositories (with compression) the default for newly initialized repos.

5 Likes

I’ve been following the spirited and detailed discussions on the issue thread and would like to thank all of you for your thoughtful consideration of how best to implement this, it fosters a strong sense of trust to see software developed using the “move slow and make things work” mantra. @MichaelEischer you appear to have been working away ceaselessly and your hard work is appreciated. (Currently restoring some customer data saved by restic, as I type this).

7 Likes

Impressed to read these news! Thank you so much for all that work and this wonderful piece of software.

2 Likes

Ahhh I’m so excited to try this out! Thank you all, especially @MichaelEischer :partying_face:

2 Likes

To try out the compression PR, there’s also the option to copy an existing repository:

First create a new one using restic init --copy-chunker-params --repository-version 2 --repo2 ... and then use copy to transfer snapshots from the old to the new repository. The copy command has also received a few performance improvements and should be able to copy snapshots as fast as your network connection allows it.

8 Likes

7 posts were split to a new topic: Odd repo size with max compression?

So this is pretty neat! I copied my ~/Library/Logs folder out, backed up with restic_v1, then restic_v2 using --compress off / auto / max. I then also compressed the folder with a few standard archivers to compare with the restic repositories. About the only thing that beats restic is 7-zip -mx=9, lrzip -L 9, xz -9, zstd --22 --ultra --long, and zpaq -m5 - so just the highest levels of LZMA2, Zstandard, and ZPAQ. And they all took waayyyy longer than restic did!

9 Likes

Wow. Looks like some savings at storage providers are coming up :blush:

3 Likes

wow, that really lookgs great. OK, logs are very good compressable gut its great to see those numbers. i suppose its just the initial backup. what about a few snapshots. how will they affect repo size?

Markus

Awesome! looks like I picked the right time to get a recommendation from a friend for a cool backup software that I could use for my NAS :smiley:

Are there other super experimental changes on master right now? If not I might want to set up a restic cloud backup with this new version to have the repository in the new format with compression right away. (if that goes bad I could still recover from a local backup or the current cloud backup setup)

testing on my M1 mac has gone well so far!
a small test backup with max compression was 845MB vs 928MB auto vs 935MB off. Source Folder was 936MB. I probably picked the worst possible example though since more than 800MB of that was randomly generated binary data… forgot about that one…

2 Likes

@ Restic Team: Thank you very much, I look forward to my tests with the compression.

1 Like

Just checking: performing a restic copy from a repo at v1 to a repo at v2 will compress everything?

Yes, new data added to a v2 repo always get compressed.

1 Like

This will not work with rclone repos, I guess?

And I think restic copy --compression max will copy using max compression?

The compression support works independently from the storage backend.

It will copy all data that does not yet exist in the destination repository using maximum compression.

1 Like

Would it make sense to add a flag like --cignore "*.zip" to skip compression for .zip files, for example?

1 Like

First time using the new build on my iMac to back up a Window’s user profile after running ddrescue on a dying disk, mounting via Dislocker, and running Restic to grab the good stuff!

Total data backed up: 272.32 GiB
Total data after dedup: 179.08 GiB - 66% original size
Total data actually written to disk (compressed): 118.56 GiB - 44% original size!

:smiley:

3 Likes

Cool! This is at max compression?