How to set the Min/Max Pack Size?


#1

Is there a way to set the Min and/or Max Chunk/Pack size when Init or Backup to a Repository?


#2

There’s no way to do that without changing the code. The sizes are chosen as a compromise, originally to accommodate local/sftp backends. You can play with the constant values in the code.

What are you trying to do? What’s your use case?


#3

No worries. I’d come across a post that talked about it and it seemed like it might have been a hidden flag or something, but was probably just Forked code.

The main reason was because I have some large Repos that are made up of predominantly very large ProRes video files. These files “never” (maybe in some distant future) get deleted, and only more get added. They do however get reorganized and renamed.

Having them split up into 1-8MB pieces makes using Cloud storage quite slow and restrictive… I’m currently using Wasabi, but was hoping to migrate to using Google Drive. Google’s Team Drives have a limit of 400,000 files per drive, and all my Repos already have > 1m files. I’m not sure if Google My Drive has the same limitation, but figured it would be worth getting ahead of the growth issue in general.

Forcing the file sizes to 100-250MB a piece seemed like it would probably help a lot for my situation. Although I dunno what this would mean for the handful of smaller project/meta files that do change occasionally.

For me at least, its only the Repo that has backups from my Laptops that it makes sense having smaller chunks, because files on there change often. Most of my other content is Static.


#4

The bigger issue with large pack sizes is that even tree objects are stored in packs. When you forget some snapshots and prune, restic tries to discard all unused objects, which requires the packs containing these objects to be rewritten and consolidated.

If you have just a few large monolithic packs, prune is going to be a very painful process as it will likely have to rewrite most of the packs in the repository when you do your weekly/monthly prune.

You’d probably have to do some testing to determine the best pack size that both meets your goal of having fewer files, but doesn’t make prune operations have to download most of the data in the repository.

On the other hand, if you never delete anything, you might not really need to delete any snapshots, as you wouldn’t recover very much space, so the prune issue might be a moot point.


#5

Good point, I didn’t realize about the trees. I’ll see if I can trackdown and play with the values to see what kind of difference it makes for me.


#6

Please report back if you do! I’m interested in your experiences.