Compression support has landed in master!

Hahaha of course I finally got my repo copied over to the new format as the prune PR was merged. It took about 30 days - though that wasn’t nonstop. I had about 5.3TB of snapshots. I now have 2.4TB of snapshots (I used max compression). That’s amazing :star_struck: thanks everyone who worked on this, especially @MichaelEischer!!

I should note that my work repository is comprised mostly of Word, Excel, and PDF documents. Along with some huuuuge genomic plain text datasets - a good terabyte’s worth of just that. I work with scientists at a hospital. So possibly much more compressible data than your average user haha


Finally upgraded my repository.
Repository size was reduced from 3171GB to 2423GB.

Plus due to huge prune decided to also change pack size from 256MB to 128MB.

Thanks a lot!

1 Like

Thanks to all the contributors, this is an awesome feature! :partying_face:
I look forward to seeing this in the next stable!

Here are my results (new repos, max compression) from tests with small datasets

  • dataset 1 (mix of text / various files): saving -50% :grinning:
  • dataset 2 (mostly non compressible files): saving -5%

Time needed for compressed compared to uncompressed: +30%
(while working with computer, not a problem for me if space savings are good)

Questions for migrating a big repo (500+ GB)

  1. where can I see if compression is active for repo?
  2. what do I need to migrate repo to new format and compress existing repo?
  3. how much space is needed for migration? or is it done in place?
  4. does migration need a lot of RAM or just time/cpu?
  5. can I continue migration if e.g. disk runs out of space?
  6. is it possible to use repo for backups during migration?