Concurrent backups to a single repo possible?

I’d like to deploy restic as d2d backup solution in a local network with several Linux servers. There should be one backup server holding the restic repository, all other servers should backup their data automatically to the repository on the backup server. Some data needs to be backed up more frequently than others (say, hourly vs. daily), this should be managed automatically by cron jobs.

I plan to use rclone for implementing an append-only strategy (for security reasons, backup clients must not delete or modify existing backups)

I have a test setup now running for some time.
On the backup server I have a repository located under /data/backup/restic/repo. The backup server is running CentOS 8 with restic 0.9.6. The backup clients run various different Linux distributions with restic 0.9.6.

For all backup clients I have their SSH public key stored in the $HOME/.ssh/authorized_keys file of the restic user on the server like this:

restrict,command="rclone serve restic --stdio --append-only /data/backup/restic/repo" ssh-rsa AAA...

On the backup clients I set the following environment variables:

export RESTIC_REPOSITORY=rclone:
export RESTIC_PASSWORD_FILE=/etc/restic/repository.pwd

and then I use restic in backup scripts like this:

restic -o rclone.program='ssh $RESTIC_SERVER_ADDRESS' backup /dir

This command then creates a new snapshot in the repository with the host set to the backup client host where the script is running on and the path set to the given directory.

Though this setup works in principle, the backup always seems to fail if backup jobs on different clients are started concurrently at the same time.

I.e. if I have a long running backup job on one client and then start another backup job on another client (both backing up to the same repository on the same server), the jobs just seem to hang as soon as the second job is started.

So far I can only have reliable backups if I make sure to start all the backup jobs on all the backup clients in a strictly sequential, non-overlapping order. Needles to say that this is very hard to achieve in a real-world scenario…

So the question is: does restic support several concurrent backup jobs to the same repository when using SSH/rclone at all?

Thanks for any insight!

– andreas

I’m just curious, why don’t you use rest-server directly instead (along with its --private-repos and --append-only)?

Also, restic 0.10.0 was released yesterday, I highly recommend using that over 0.9.6 as it contains a lot of improvements.

Multiple concurrent backups to a single backup repository should just work.

Depending on the amount of data you backup it might be a good idea to use separate backups for different clients.

the jobs just seem to hang as soon as the second job is started.

Could it be possible that the first job is cancelled for some reason? restic versions before 0.10.0 tend to just hang if the rclone connection exits unexpectedly, so restic 0.10.0 could help with assessing whether that is the case. (Or just check the running processes ^^ )

1 Like