Compression support has landed in master!

Hahaha of course I finally got my repo copied over to the new format as the prune PR was merged. It took about 30 days - though that wasn’t nonstop. I had about 5.3TB of snapshots. I now have 2.4TB of snapshots (I used max compression). That’s amazing :star_struck: thanks everyone who worked on this, especially @MichaelEischer!!

I should note that my work repository is comprised mostly of Word, Excel, and PDF documents. Along with some huuuuge genomic plain text datasets - a good terabyte’s worth of just that. I work with scientists at a hospital. So possibly much more compressible data than your average user haha

5 Likes

Finally upgraded my repository.
Repository size was reduced from 3171GB to 2423GB.

Plus due to huge prune decided to also change pack size from 256MB to 128MB.

Thanks a lot!

1 Like

Thanks to all the contributors, this is an awesome feature! :partying_face:
I look forward to seeing this in the next stable!

Here are my results (new repos, max compression) from tests with small datasets

  • dataset 1 (mix of text / various files): saving -50% :grinning:
  • dataset 2 (mostly non compressible files): saving -5%

Time needed for compressed compared to uncompressed: +30%
(while working with computer, not a problem for me if space savings are good)

Questions for migrating a big repo (500+ GB)

  1. where can I see if compression is active for repo?
  2. what do I need to migrate repo to new format and compress existing repo?
  3. how much space is needed for migration? or is it done in place?
  4. does migration need a lot of RAM or just time/cpu?
  5. can I continue migration if e.g. disk runs out of space?
  6. is it possible to use repo for backups during migration?

Thanks!