I’m using restic in my home setup of quite a few Raspberries and a central server running a restic rest server.
All of them are up 24/7, my desktop (the largest machine) uses restic to backup data to an external USB-hard-drive (and probably to an offsite server in the near future), but it is up only a few hours a day.
I’m looking into copying over the backups from the rest server to the desktop machine’s external drive every day (when it’s up). I’m aware of the copy feature in restic backup, but the overhead of unpacking/re-crypting seems high (all my backups share the same password as they are on-site), de-duplication would not be a major advantage, so it need not be shared between repositories.
First I thought I’d run a systemd job to check the repositories on the rest server and copy them over using rsync, but on second thought it seems strange to go through 2 different methods. Also the backups of my Raspberries may be in progress at the same time and I don’t want to add extra coordination.
So …
Is there a simply way to copy the raw data from the rest server in a coordinated way (not interfering with a running backup, heeding a local lock)? The Raspberries typically only take a few seconds to perform their backup, so any potential delay wouldn’t be a problem.
If that makes no sense (quite probable), how do I make sure the newly copied backups don’t potentially destroy a valid copy on the external drive.
And …
The obvious solution to move the hard-drive to the server running restic-rest is not valid as its current location in the house is selected for panic removal (in case of fire etc.)
Just keep it simple and schedule your RPi backups in such a way that they leave a time window during which you can rsync or rclone the data from the rest-server to the USB drive?
The point is that timers may be expired when the desktop machine boots, at whatever time. I could check the time in the sync-script and delay until the RPis are surely done, however.
In general I don’t really like this kind of assumption in a distributed environment, it’s currently 8 RPis and growing …
Well, considering that restic when backing up only ever adds data to the repository, I would say that it’s safe to grab a copy of a repository that is being backed up to.
If you do that and the running backup is e.g. 50% done, then your copy of that repository will simply look like the running backup was interrupted at 50% and you then made a copy of the repo.
And the next time you rsync/rclone the repository, you will get the rest of the backup that was running when you did the syncing (including the snapshot).
Probably that is good enough for the synchronization issue. The central server runs a prune process once a month, which might generate another headache …
Additionally I was a bit concerned about integrity of the backup under the circumstances, that the server messes things up, e.g. local driver error, but I can probably handle this with the pre-check (as described).
I might end up with a second restic rest server on the desktop machine and a script to trigger the backup of the backups from the server to it, which should address the synchronization and potential data error issues, but obviously not the performance impact.
From my perspective the safe duplication of backups would probably be worth a more general discussion.
If you want two copies of your backups, then just run two backup jobs on each client. One to your first rest-server, and the second one to your other rest-server.
Might well be, but simply performing 2 separate backups, won’t work as the second server is not up all the time. So I need a way to trigger it from there, but on the other hand I want to avoid this kind of central management.
Anyway, I’ll stick to the rsync method and some wiggle-waggle to avoid inconsistent state issues.
Thx for your help.
Well, I mean, you can without too much fuss set up a configuration where by some scripting or similar triggers backup runs on the clients whenever that second backup server is up. But do whatever floats your boat.
I finally followed rawtaz suggestions, at least in a way, and I’d like to present my solution. Maybe someone runs into the same kind of situation/confusion.
I duplicate the backup by activating 2 different backup scripts on each individual node (mostly RPis), one that goes directly to the 24/7 server (which should always work) and the second which polls the desktop machine using:
‘neo’ being the desktop machine ‘morpheus’ the repository for the machine performing the backup
The script will run until the desktop machine shows up and is activated daily by a systemd timer event. The timer will not start a new instance of this process unless it has already terminated (the default behaviour).
This way all nodes in my network will back-up independent of the existence of the desktop machine, but will always create a second backup on it if possible. Also checking, forgetting and pruning is duplicated that way and should not create any (additional) inconsistencies, there is no additional need to synchronize nodes.
Edit: You have to make sure a backup on the restic rest server is created once, otherwise the config file for the system won’t exist and 404 is returned (and the backup will never be started). Simply start the backup by hand once when the machine is up.
