Compression support has landed in master!

Development
41

Hahaha of course I finally got my repo copied over to the new format as the prune PR was merged. It took about 30 days - though that wasn’t nonstop. I had about 5.3TB of snapshots. I now have 2.4TB of snapshots (I used max compression). That’s amazing :star_struck: thanks everyone who worked on this, especially @MichaelEischer!!

I should note that my work repository is comprised mostly of Word, Excel, and PDF documents. Along with some huuuuge genomic plain text datasets - a good terabyte’s worth of just that. I work with scientists at a hospital. So possibly much more compressible data than your average user haha

6 Likes
42

Finally upgraded my repository.
Repository size was reduced from 3171GB to 2423GB.

Plus due to huge prune decided to also change pack size from 256MB to 128MB.

Thanks a lot!

1 Like
43

Thanks to all the contributors, this is an awesome feature! :partying_face:
I look forward to seeing this in the next stable!

Here are my results (new repos, max compression) from tests with small datasets

  • dataset 1 (mix of text / various files): saving -50% :grinning:
  • dataset 2 (mostly non compressible files): saving -5%

Time needed for compressed compared to uncompressed: +30%
(while working with computer, not a problem for me if space savings are good)

44

Question. When running restic prune without --repack-uncompressed, does the --compression flag have any effect? For instance, if I’ve repacked my entire repo with --compression max and then I periodically run a normal restic prune - is the small amount of data it’s repacking being repacked with auto compression instead of max? Not gonna lie, I wish there was a way to set the default compression level for a repo haha

45

The data would be repacked using the auto compression level unless something else is specified. The repack step first decodes and verifies blobs before moving them to a new pack file. That also implicitly requires the blob to be compressed again.

1 Like
46

BTW: rustic allows to save the zstd compression level (which can be between 1 and 22) in the config file, see Add config command by aawsome · Pull Request #76 · rustic-rs/rustic · GitHub

1 Like
47

Great feature!

Would you mind adding an ENV variable for the compression parameter?

Something like RESTIC_COMPRESSION would be nice.

3 Likes
48

I’ve opened PR Add environment variable RESTIC_COMPRESSION by MichaelEischer · Pull Request #3841 · restic/restic · GitHub to add an environment variable.

3 Likes
49

Cool stuff, thanks for the PR!

50

Hi everyone,

I am testing Restic to backup my project and it seems promising (incremental, encryption and compression).

I see on this thread the compression option should be available on master since April 30th 2022.
I am using v0.13.1 from ubuntu 20.04 and the self update (done today) :

restic 0.13.1 compiled with go1.18 on linux/amd64

I see no option for repo version 2 and compression mode :

restic init --repository-version 2
unknown flag: --repository-version

restic init --compression max -r
unknown flag: --compression

Am I missing something to enable compression ? (which is mandatory for my project)

Thanks in advance,
Arnaud

51

0.13.1 is not current master.

The self update will update to the latest stable release.

To use a nightly/master build you need to install from a nightly tarball or build from source.

52

Thank you for your answer.
I am going to try.

Do you know when compression will be available on the stable release ?
(To use Restic on our production environment)

53

0.11, 0.12 and 0.13 have been released with roughly 9 months between each other. So 0.14, which would contain the compressed store would be due Christmas, roughly.

Though that’s an extrapolation.

In reality I think the new repository format needs to be released with a lot of care, to avoid having to publish a third version after that quickly.

So as little as I like it, as much I have to say, I don’t think there will be a new release before 2nd quarter of '23.

54

The new format already seems to be quite stable. At least there have been very little problem reports so far. So I’m pretty confident we’ll be able to release the next version this year.

7 Likes
55

Slightly better compression ratio than BorgBackup on a test repo of around 50 GB with default compression.

Any comparison with ZFS? Probably better compression than ZFS.

56

It uses zstd with 512 KB windows if I’m not mistaken, and you can set compression to max if you like which will set the corresponding option for zstd. Don’t need to benchmark backup solutions’ compression separately, just look at what compression they actually use :slight_smile:

57

Yeah they both use lz4 which is slightly worse than zstd used by restic in compression ratio.

58

Try again with zpaqfranz and just with default settings. Not zpaq and not with -m5 option. Filesize should be very close to or under 50 MB and speed way faster than anything else. Even with the CRC checksums that zpaqfranz has built in.

59

Sure thing. Copied my latest log folder out, and ran some more tests:

Screenshot 2023-08-06 at 11.49.15 PM
Screenshot 2023-08-06 at 11.49.15 PM658×1574 189 KB

Sample of results near zpaqfranz -m1, sorted larger to smaller:

restic_auto, 3.600s, 127.9MB
xz_1, 15.271s, 121.8MB
zstd_6, 2.198s, 120.8MB
restic_max, 6.721s, 120.3MB
zstd_7, 2.820s, 120.1MB
zpaqfranz_1, 9.016s, 119.1MB
zpaq_1, 7.743s, 119MB
zstd_8, 3.223s, 119MB
zstd_1_long, 2.318s, 111.9MB
bsc_e0, 34.241s, 111.3MB
zstd_2_long, 2.343s, 111MB
zstd_7_long, 2.934s, 105.8MB
zstd_13_long, 7.861s, 102.8MB
zstd_18_long, 62.84, 97.9MB

Sample of results near zpaqfranz -m1, sorted slower to faster:

zstd_18_long, 62.84s, 97.9MB
bsc_e0, 34.241s, 111.3MB
xz_1, 15.271s, 121.8MB
zpaqfranz_1, 9.016s, 119.1MB
zstd_13_long, 7.861s, 102.8MB
zpaq_1, 7.743s, 119MB
restic_max, 6.721s, 120.3MB
restic_auto, 3.600s, 127.9MB
zstd_8, 3.223s, 119MB
zstd_7_long, 2.934s, 105.8MB
zstd_7, 2.820s, 120.1MB
zstd_2_long, 2.343s, 111MB
zstd_1_long, 2.318s, 111.9MB
zstd_6, 2.198s, 120.8MB

Zpaqfranz seemed fast, though not as fast as Zstandard, of course. Interestingly enough, also not quite as fast or efficient as regular Zpaq, either.

I’m not a huge Facebook fan, but the one good thing that came out of Meta was Zstandard. So fast and efficient. Glad that’s what Restic decided to use!

60

Will running restic migrate upgrade_repo_v2 automatically compress the existing repo?