Central Management Server

Getting Help
1

Is the following setup possible with restic?:

  • Central backup server where all clients will back up to
  • Each client has their own key and also the central server adds its own key to each repo
  • Each client backs up at specific interval
  • But the central server manages a weekly purge and restic check process on each of the clients’ repos, which is possible because the central server has a key
2

Yes; a restic repo can be accessed through multiple keys (but they don’t have any permission management – your server would have to do that – and keys aren’t hierarchical).

3

Yes and it is pretty simple. Keys not having permissions makes that every key works essentially the same; there’s no master / limited access key. However, you could serve the repo to clients through a rest server with append only mode so the server is the only one who can actually delete data.

4

Yes, just use the restic REST server.

Yes, this is possible. Note that the server would have to add a key to each repository. The passphrase can be the same, but the key would be different.

The server cannot enforce this or otherwise command clients to back up.1 The clients would have to be configured to do this, for example using cron on *nixes or the task scheduler on Windows.

Yes, the server can do this since it has a key in each repository. Note that the prune/check schedule needs to be coordinated with the clients’ backup schedule, since prune and check both require an exclusive lock on the repository. If the server has the repository locked when a backup starts, the backup will fail. (Hmm, @fd0, would it be possible for restic backup to retry locking with backoff so that backups are delayed instead of aborted?)

1 Technically, this can be done, but it’s not supported as a native restic feature. For example, you could have the backup server command a client to run a backup script over SSH.

5

To deal with this, I’ve been using loops like the following:

LOCK_WAIT=1800    # total time to allow for operation to start
LOCK_DELAY=60     # delay between attempts

EXPIRE=$(( SECONDS + LOCK_WAIT ))
until [[ $EXPIRE -le $SECONDS ]] || eval restic forget $RESTIC_ARGS $RESTIC_FORGET_ARGS; do
        echo Waiting for lock.
        sleep $LOCK_DELAY
done

Granted, this would try again on any failure that caused restic to return a non-zero value and does not provide a backoff, it has working well for me.

6

@westphaldp Unless I’m missing something, the eval in that script is redundant.

7

@cdhowie I didn’t include what RESTIC_ARGS and RESTIC_FORGET_ARGS are defined as. They may include variables that can’t be evaluated until this point. eval is handling that.

8

Very interesting. Thanks everyone for your replies!

@westphaldp So is there a way I could modify the script to only run restic backup if you get specific error code that lock is in use?

9

From a shell script, it is possible if restic returns clearly defined exit codes. I’m not sure what the state of that is. @cdhowie, do you know what restic's exit codes look like?

Otherwise, it would require processing the text output of restic to determine why it failed.

10

I would guess based on this open issue that right now you can’t tell failures apart:

11

@westphaldp Did you use LOCK_WAIT=1800 in your script because restic considers a lock to be stale after 30 minutes?

12

@Den I wasn’t aware of any lock timeout, nor have I noticed that behavior on my systems. For my timeout, 1800 seconds just seemed like a good place to start. It can vary depending on system/repository.