About restic local cache size

Features and Ideas
1

Hi again :waving_hand:

(Sorry for a wall of text :grin:)

After a routine pruning of a large repo, I saw the local cache size explosion, again. Since I was one of the people who complained about cache size for a while (1,2,3,4,mine), I thought why not take a stab at what is going on, and started digging.

Turns out, just like people repeatedly said in the replies, it was mainly related to client count in a repository :sweat_smile:. When we do prune (and allow some repacking within) restic tries to consolidate partly-used packs together and if you have a lot of data+hosts, this simply means single client’s data can spread to a lot of packs over time. The cache problem in big repos is just a representation of this fragmentation: Client may need to pull many cache-able metadata packs related to its current process on backup, thus end up with a big cache.

Then I dug into how restic does the repacking, and as far as I understand it was arbitrary. Probably due to speed concerns or maybe even randomized as security-enhancement?

What if we could affect that? Since users (at least some of them) know what they care for parent snapshot selection while backing up, we can give this as a “hint” to the prune operation. What I mean is group-by attribute:

~> restic backup -h | grep group-by
  -g, --group-by group                         group snapshots by host, paths and/or tags, separated by comma (disable grouping with '') (default host,paths)
      --parent snapshot                        use this parent snapshot (default: latest snapshot in the group determined by --group-by and not newer than the timestamp determined by --time)

By default we have host,paths as selection criteria, and this is not utilized on prune (afaik). So I created a new branch to optionally pass --group-by values to prune too, so repack step can create “groups” which should be used to bundle “related” metadata closer.

Of course this will not help everyone, especially if the clients are (allowed) passing different --group-by values (but worst case it might be as bad as current “random” selection we have, or?). In managed environments like ours the gains were immediate: 7+ GB cache usage on each client dropped to 1.4GB after first prune run with this change. Second run caused less than 500MB of cache :exploding_head:.

You can find the change here.I put it behind a feature flag and disabled it by default, so nothing changes for a “normal” prune operation. But I thought this might be an interesting case to contribute to upstream? Even if it’s not, I thought this was worth sharing for other gigantic repo owners to understand what is going on :sweat_smile: With this at least I can trigger a “special repacking prune” once in a while to cut the client cache usage.

Disclaimers:
a. Even though I’ve coded fair amount of time, I am not a “developer”, let alone a golang developer. This whole investigation and code snippets were made with help of LLMs.
b. I tried my best to understand and make it as painless as possible for all parties, it is not a trivial change, so only try this at “home” as much as you trust a random internet user’s untested code!
c. Also sorry if there was any development on that front already, I might be victim of the old quote: “A month in the laboratory can often save an hour in the library” :smile:

2 Likes