Repack smallest packs

Getting Help
1

In my repository I have different pack sizes: ~ 4 MB (probably an old default?), ~16 MB (the current default) and ~ 64 MB (set by me). The repository is 1.8 TB and has over 220k packs.

I would like to repack the 4 MB packs to 64 MB (because there are so many) but leave the 16 MB packs as they are (because that’s also a reasonable size). I guess restic prune --repack-small will repack all below 51.2 MB (80 % of current pack size setting). So I am looking for something like --repack-smaller-than 14M.

Is is possible to have a repack threshold lower than the target pack size?

restic-pack-sizes
restic-pack-sizes1854×965 35.3 KB

2

restic prune --repack-small --pack-size 16 should do the trick for you.

3

But it will repack to 16 MB, won’t it? I would like to repack to 64 MB but just those ~ 4MB packs.

4

You are right. I misunderstood your question. Sorry. :neutral_face:

5

Looking at the repack logic at prune.go, more precisely at decidePackAction(), there is a comment that states that smaller packs are repacked first.

Maybe some more experienced users can confirm, but it seems that if you run restic prune --repack-small --pack-size 64 --max-repack-size X, where X is the size in MB of all your < 16 MB packs together, you should be able to do what you want.

1 Like
6

Thanks for looking that up! I figured out that the smallest packs are the oldest from a v1 repository (uncompressed). So if I use --repack-uncompressed, the smallest packs disappear and new 64MB packs are created, because the oldest packs are uncompressed and small. I am already using --max-repack-size because it would take too long to repack all in one go (about 30 hours for all and perhaps 15 hours for those < 10 MB).

Good to know that restic starts with the smallest packs!

1 Like
7

@bazinga Where did you find that comment? restic/internal/repository/prune.go at 33da501c357050a8b492342daa8e3bb403ed2bbd · restic/restic · GitHub is the closest I can find, but that only states that too small pack files are sorted before large enough pack files. There is no specific sort order among too small pack files.

1 Like
9

Hmm yeah besides that line, all I’m finding is:

327: only repack very small files by default
445: Sort repackCandidates such that packs with highest ratio unused/used space are picked first.

I could see someone misreading 327 as “smaller packs are repacked first” perhaps?

10

That’s the comment that I interpreted as “smaller packs are repacked first”. Sorry about the confusion.

11

That was the comment on line 449:

// Moreover packs containing trees and too short packs are sorted to the beginning

12

Well, thanks for the clarification. That comment, while being technically correct, can be easily misunderstood. If I understand the code correctly, when using --repack-small small packs (<80% target pack size) are repacked like 100% unused packs. So in my case 16 MB packs are repacked while 99% unused are not.

In my particular case, using --repack-uncompressed works pretty well. But IMHO, a --repack-smaller-than <size theshold> flag would be a nice feature for repository management.

13

Please open an issue on Github, we can’t keep track of feature requests on the forum.

14

Here it is: Selection of packs to repack based on size · Issue #5109 · restic/restic · GitHub

1 Like
15

For the sake of completeness, here is the new pack size distribution and mean pack size is now 41.4 MB. Most small packs have been repacked to 64MB, the 16 MB packs remain.

Figure_1
Figure_11854×965 31.1 KB

The set of small packs was not completely identical to the set uncompressed packs, so there are a few small packs left which I can only repack if I repack all the ~16MB packs, too. The log scale shows them:

logscale
logscale1854×965 31 KB