What are the implications of storing data from different hosts in one repository?

Getting Help
1

Hi @fd0
We are trying to understand the implications of storing data from different hosts in one repository. I am mainly using remote backups. Here are some questions:

  • Restic takes a lock when taking backup. Can I backup 2 hosts at the same time to the same repository? I know that I can pass --no-lock to avoid locking. Do you see any problem with that approach?

  • It seems that I can apply retention policy independently to each host. Can you please confirm?

  • In case of a disaster, will it be faster to recovery if I use separate repo per host vs one repo for all hosts?

  • Are there any other safety concerns with repository?

  • Say I have cluster with 100 nodes. They can potentially all start backup at the same time. Are there any scaling concerns between separate repo per host vs one repo for all hosts?

Thanks.

1 Like
2

You can backup two different hosts into the same repo at the same time, that works very well. Please don’t use --no-lock except for restoring/accessing repos on read-only locations, that’s what this parameter is for.

When you run restic forget or restic prune (which both may remove data), it’s very important that no other restic process is creating new data in the repo which references the data that is to be removed, so those two operations need to run without any concurrent restic processes.

Sure, that’s easy to do, just pass --host to restic forget and it’ll only consider snapshots from that host (and ignore the others). Please make great use of the --dry-run option, which will simulate what happens for a given policy :slight_smile:

Shouldn’t matter at all, the speed will be roughly the same.

Run restic check --read-data regularly, but be aware that it’ll download ALL data (which may be expensive). If you can run commands on the repo server, you can additionally use sha256sum on the files, and compare the resulting hash with the file name to detect bit rot. You can also use a small program I wrote to automate that: GitHub - fd0/psha

Not for backup, no. But while restic prune is running (which cleans up unreferenced data), no backups can be made. That’s a limitation we currently have. So you need to carefully schedule that. And it make take some time…

4 Likes
3

@fd0 Can you confirm that a prune prevents other hosts from running a backup? Or is it simply that they should be prevented (by the end user) from running during this time?

4

If you don’t use --no-lock option (for both backup and prune), then backup will fail with something like this:

unable to create lock in backend: repository is already locked by PID 18813 on HOSTNAME by restic (UID 119, GID 65534)
5

Running prune blocks the repo for all other clients, for all other activities such as backup. It’s a safety feature, all commands which may remove data (prune and forget) require an exclusive lock on the repo.

We have several ideas on how to change the repo format and the design a bit to allow concurrent backups while pruning (a use case often requested), but that will take some time to implement. For the first iteration of restic’s prune command we decided to play it safe and not allow anything concurrently.

3 Likes
6

Excellent - that’s exactly what I was hoping would happen :slight_smile: Always better safe than sorry! I’d been running prune, but not stopping my other servers scheduled backup tasks, so was hoping this wouldn’t have had any adverse affects. Thanks for the confirmation!