I was wondering if anyone had thoughts about this non-traditional “distributed computing” use-case for restic, please.
This is the use-case:
A single restic repository in S3
3 WRITERS (each running “restic backup”, concurrently on different servers that all mount the same shared CIFS drive, which has 1 million files)
100 READERS (each running “restic restore”, concurrently from different servers, and each restoring only a relatively small number of files, perhaps 10 files each only, from the last available snapshot each sees)
WRITERS run about every hour (24h), to backup the entire shared drive, but only a random 1% subset of the 1 million files are actually modified daily on average (by a completely separate process not described here) so the incremental backups are not “large”
READERS run much more frequently (e.g, think every minute on average) and the 10 or so files that each of them gets can be thought of as being a random subset of the 1 million each time they run (in fact, these READER servers are responding to “requests” submitted by hundreds of independent users of the distributed computing application)
There is no pruning/deleting whatsoever in the repository (we keep all snapshots)
I understand that WRITERS might possibly lock out other WRITERS when/if they acquire exclusive locks and I am not concerned with this.
My questions are:
Can READERS ever be locked out by WRITERS?
Can WRITERS ever be locked out by READERS?
Can READERS ever be locked out by other READERS?
Of course, since there is no pruning/deleting in our use-case, it is not hard to achieve a design in which the answer to all 3 questions would be NO. For example, by using immutable “persistent data structures”. Our question is whether it is possible to use restic in this way today, e.g., would restic restore –no-lock achieve exactly this? I am sure you understand that my question is not about the probability of such locking but about the mere possibility based on the inherent design.
Would love to hear your thoughts!!! Thank you so much and happy restic’ing!
Locks in restic are either shared or exclusive. Shared locks don’t conflict with out shared locks. Only exclusive locks conflict with other exclusive locks or shared locks. That is there can either be one exclusive lock or an arbitrary number of shared locks at one point in time. This somewhat similar to the classic ReadWrite-Locks available in many programming languages. However, the terms “read” and “write” don’t match the semantics of lock in restic.
Operations that read data or add data to the repository use shared locks. Removing data or checking the integrity requires an exclusive lock. The “READERS” and “WRITERS” you’ve mentioned only require shared locks, so the answer to all three questions in no. There’s no need for restore --no-lock etc. When a large number of locks are active at the same time, then the check whether a repository is locked can be quite slow, see also Many restores lock · Issue #3652 · restic/restic · GitHub .
Regarding the overall description I see a few long-term problems. Each reader has to fetch all current indexes from the repository before doing anything useful. Unless you’re using restic mount this has to happen for each individual restore from the repository. The number of indexes and their size will grow over time which will take longer and longer to load. The same problem also affects snapshots. When telling restic to restore from the latest snapshot, then it has to load every single snapshot to find out which one is the most recent one.
The overall description sounds a bit like you actually want a distributed file system, but I can’t tell whether these can store their data in S3.
Though it may not be a perfect fit for the reasons you mentioned, a benefit of using restic here is that it supports maintaining a copy of a file system on S3 out of the box, and it is very good at it! Another is that, when such consistency is desired, all the sub-tasks of a particular distributed job can use the same snapshot of the data (the snapshot identifier can then be passed as input parameter to each READER executing a sub-task), which is not supported by every distributed file system.
In my previous post, an assumption was “no delete whatsoever”. If now there was a “short” window of time (daily, weekly or monthly) during which READERS are “turned off” to allow the system to run forget of older snapshots, followed by prune, rebuild-index, and integrity checking; then I assume the system would maintain the same performance characteristics over time (1% of data changes daily and 1% of files gets added/deleted monthly but the total number of files remains about 1 million). Is this correct? Of course the trade-off is then the duration of that window of downtime. For that, I don’t know what to expect and that’s where I see the most operational risk and project planning uncertainty.
If the number of snapshots does not grow without bounds then the performance should remain somewhat similar. There may be a certain (probably small) degradation of the restore performance over time as the file parts tend to get less ordered over time. However, I have no performance numbers of any kind here. It’s just an effect that seems to be common for deduplicated storage.