So this is as much a thought experiment or theoretical question as it is an actual specific use case, but imagine that I have a machine that I want to backup, and this machine has a relatively bi-modal distribution of file sizes: by which I mean that there a large number of relatively small files, and a large number of relatively large files, but not all that much in the middle. The theoretical example I got to here was "I want to back up a ton of very small config/source files from /etc and /home, but also back up a ton of large video/image file.
I got as far as “restic appears to support different pack sizes within the same repo” – so I thought about a setup where I use one command “pack-size=XXX” for the location full of mostly small files, and “pack-size=YYY” for the source locations with the big media files. (Here, YYY is presumably much larger than XXX… I’d have to think a little bit about the exact sizes but doesn’t change this question).
Does this sound like SHEER GENIUS? (unlikely) … or does it sound like “Hmmm, that actually makes sense.” or does it sound like “That’s stupid and doesn’t buy you anything, and here’s why…”
mixing pack sizes in one repo is fine. restic already stores packs of whatever size each backup run used. it wont break anything.
does it buy you much for small configs vs big media though? probably not that much. --pack-size just controls how chunks get bundled into pack files for upload. the chunker is what cares about file contents, and that stays the same for the whole repo.
so you can run a smaller pack-size for /etc and a bigger one for video paths if you want. its more “harmless, maybe slightly nicer object counts” than sheer genius. for me id keep one pack size and not complicate the scripts unless youve measured a real problem.
separate repos only if you want different retention or keys. not just because file sizes are bi-modal.
@ljwobker What is actually the problem you like to solve? Note that sizes of the files are not equivalent to pack sizes in the repository. resic chunks large files into smaller pieces and packs small pieces into pack files - so it by default is able to treat both very small and very large files with a single pack size “configuration”. What bothers you about this default behavior?
As I said, it was more a thought experiment and a “how does it work” question as opposed to solving or optimizing for a specific thing… thanks!
@ljwobker i’d focus on the documentation hints:
- Use larger pack size to reduce the number of files in the restic repository. Sometimes required for large repos at backends which have a file quantity limitation. If you mix up pack sizes then you are cancelling that effect.
- Use larger pack size to improve upload performance by sending fewer bigger items. Note the mentioned disadvantages of larger local files space requirements.