Share cache for multiple hosts via NFS


is there anything host specific in the restic cache that makes sharing the cache between different hosts a bad idea?

Long story:
I want to switch some systems to restic where i expect the cache size to be a problem. The setup is that on one physical host running kvm a lot of small, nearly identical virtual machines are running. So both the amount ouf RAM and the amount of storage is limited.

So the plan is:

  • backup all vms to the same restic repo
  • have a script running on the host that runs the backup in sequence for each vm
  • increase the amount of RAM before running restic
  • put the cache dir on an NFS exported file system on the host
  • prunes will run on the host

I did some small tests and that seems to work great (only slightly slower than lokal cache, cache size increases only slightly with each new host) but i am unsure if sharing the cache will lead to any problems later.

hello @Ramo,
i am little confused about your scenario as you mention in the title “share cache for multiple hosts” and then you mention “one physical host”. In the end you have “each new host”.
Is my assumption correct that you have multiple hosts with each kvm + VM?

I cannot answer your question about the content of restic cache and if sharing is a bad idea. But I have two suggestions.

Are you restricted in the NFS filesystem size? Else I would maintain one cache folder for each host and avoid sharing the cache.

Did you try or consider using no cache at all:

> "The parameter –no-cache disables the cache entirely. In this case, all data is loaded from the repository."

The backup job would run long(er) on the physical host.

@ramo Where is the repository located? A cache only makes sense if it is quite a bit faster than the repository itself, of course :wink:

Besides this - in general sharing a cache shouldn’t be a big problem. I don’t know exactly how concurrent writes to the cache would play out (depends on how your NFS is behaving) but in general there shouldn’t be concurrent writes of identical files if all clients of a repository share the same cache.

Hi thanks so far, i think i will give it a try.

To answer your Questions:

  • I have multiple (physical) Servers but each will be set up the same way so that will not matter.
  • Each physical Servers runs multiple virtual machines (between 10 and 35 currently)
  • The repositories for restic are two machines in the same network running restic-rest-server
  • The nfs share is served by each physical host for his virtual machines (aka fast…)

in the meantime i did some initial measurments in a test env:

  • Time for a restic backup with local cache: ~ 2min
  • Time for a restic backup with cache on NFS: ~2:15min
  • TIme for a resic backup using --no-cache: ~14min
  • Size of cachedir on nfs after backing up first virtual machine: 900M
  • Size of cachedir on nfs after backing up second virtual machine: 930M

@ramo, so I conclude you run your backups from within the VM to the rest-server. Interesting scenario compared to snapshotting/backup the VM from the physical host.

Your tests show that performance-wise it works well !

Personally i would investigate if you can switch from running a NFS server on each physical host to share a folder to each VM, towards directly mounting the host restic-cache folder into each VM. That would simplify the setup and reduce overhead.

With docker and .yml files that is very easy to do.

Similarly, this allows to mount the folders of your VM which contain the data that you want to backup onto the physical host. Then you can run restic on your physical server to back up that data to the rest-server. I do something like this at home where i run stock containers and backup the config and data at the host via these mounts. Not sure if this fits in your VM scenario.

Sharing the cache directory should work fine with recent restic versions (>= 0.15.0).