Bash script to back up to 2 different repos

so, I’m running a bash script to backup to a local USB drive using

export RESTIC_REPOSITORY=/…/USB-Backup/backup
export RESTIC_PASSWORD=…
/usr/local/bin/restic -r $RESTIC_REPOSITORY backup ~/thisandthat
/usr/local/bin/restic -r $RESTIC_REPOSITORY forget --keep-last 3 --prune

Now I’d like to backup the same directories to another mount. Any idea if I can just add a new line (I’d use the same PWD, like

    export RESTIC_REPOSITORY=/…/USB-Backup/backup
    export RESTIC_REPOSITORY=/…/2ndDIR/backup
    export RESTIC_PASSWORD=…
    /usr/local/bin/restic -r $RESTIC_REPOSITORY backup ~/thisandthat
    /usr/local/bin/restic -r $RESTIC_REPOSITORY forget --keep-last 3 --prune

or rather
export RESTIC_REPOSITORY=/…/USB-Backup/backup /…/2ndDIR/backup
What do you think?

No, you can’t do that. The second line will simply replace the value in that variable, as if the first line was never there.

Nor can you name two paths in the same variable, because that’s not how restic is expecting the repository to be specified (it only expects one repository to be listed).

Simply move the RESTIC_PASSWORD line to the top, then duplicate the three other lines and adjust their contents accordingly.

That said, I’m not sure it’s a great idea to prune every time you have run a backup. Seems a bit excessive to me.

Thanks: I’d need to create a 2nd script for the 2nd repo, you say, right?

But why putting the pwd on top?

No, just do what I said within the same script. The script is just there to run your commands, so unless you have specific needs you can keep them all in one script.

Because it makes the code cleaner that way. It makes no practical difference.

OK, I did that; now get
Fatal: unable to open repo at /media/…/Backup: ReadDir: open /media/…/Backup/keys: permission denied
There’s a key in the keys folder though

Well, the system is telling you that you don’t have permissions to access that folder/file. You need to fix those permission issues.

dealing with it now, never encountered this before. But it’s some time since I last worked with restic.

Note that this permission issue is outside of restic though, it’s regular permissions in your filesystem that it’s complaining about.

I know, I did init as root, that was stupid. I think I sorted this out. Now checking if the 2nd backup was created…

One possible solution would be to use a loop.

repos="/…/USB-Backup/backup /…/2ndDIR/backup"
export RESTIC_PASSWORD=...

for repo in $repos; do
  /usr/local/bin/restic -r "$repo" backup ~/thisandthat
  /usr/local/bin/restic -r "$repo" forget --keep-last 3 --prune
done

This particular syntax requires that none of the repository paths contain spaces. There’s workarounds in the event that they do.

3 Likes

This seems to work quite nicely, Thanks. I now need to check if I can mount the backups I sent to SIA.

But keep in mind that if you use a loop like this, your data could change in the time the first backup starts and finishes when coming to the second repository. If that is something you can live with, carry on :slight_smile: If not it would make sense to use some sort of snapshot like in BTRFS and the likes.

Note that this applies whether you are running two consecutive commands or using a loop (which is just decoration for consecutive commands).

Using snapshots is preferred for other reasons as well, such as making sure a single snapshot is internally consistent. For example, if directly backing up a database server’s data directory, you must backup a snapshot (on stop the server) or the backed-up data will almost certainly be corrupt. We use LVM snapshots on our servers to take backups of running services without interruption.

Hey Everyone,

If anyone’s interested, assuming you have enough resources (Memory + CPU) and you have plenty of throughput to/from your storage, we can expand on @cdhowie 's idea a little to make the backups run in parallel.

This assumes that you have your restic password exported somewhere in the file, and that it’s the same for each remote. It can be modified to work around this. (Ask me if you want help with that. If you do, please let me know if you want to export the passwords in the file or prompt for them.) It also still requires each remote to not contain any spaces.

Here’s the basic idea (Based on cdhowie’s idea):

#!/bin/bash
# The Local Directory to Backup.  CHANGE THIS TO WHATEVER YOU WANT TO BACKUP!
export LOCAL_DIR="~/whatnot"  #CHANGE ME!
# REPOS - The Repositories to backup to, separated by spaces.  Remote names can't have a space in them.
# CHANGE THIS TO WHATEVER REMOTES YOU WANT!
export REPOS="remote1:directory remote2:directory2" # CHANGE ME!
export PROCS=''  # DON'T CHANGE!  - Needed to Sync the threads at the end

# Function backup_one backs up the location "$LOCAL_DIR" to the remote given as the first argument.
# The code below will call this automatically
backup_one() {
  /usr/local/bin/restic -r "$1" backup "${LOCAL_DIR}"
  /bin/sync
  /usr/local/bin/restic -r "$1" forget --keep-last 3 --prune
}

# Loop over the list of repos, spawning a new thread for each one
for repo in $REPOS; do
  backup_one "$repo" &
  export PROCS="$PROCS $!"
done

for proc in $PROCS; do
  wait $proc
done

Hopefully this helps someone.
Good luck all,
jedi453

1 Like

If I’m not mistaken, you can replace this with a single line: wait (and also remove the $PROCS variable). wait by itself should wait for all background jobs.

Side note, you don’t need to export any of these variables. Exporting a variable adds it to the environment of commands that are executed within the shell, but none of these variables are used outside of this shell script.

We can simplify the script:

#!/bin/bash

# The Local Directory to Backup.  CHANGE THIS TO WHATEVER YOU WANT TO BACKUP!
LOCAL_DIR="~/whatnot"  #CHANGE ME!

# REPOS - The Repositories to backup to, separated by spaces.  Remote names
# can't have a space in them.
#
# CHANGE THIS TO WHATEVER REMOTES YOU WANT!
REPOS="remote1:directory remote2:directory2" # CHANGE ME!

# Function backup_one backs up the location "$LOCAL_DIR" to the remote given as
# the first argument.  The code below will call this automatically.
backup_one() {
  /usr/local/bin/restic -r "$1" backup "$LOCAL_DIR"
  /bin/sync
  /usr/local/bin/restic -r "$1" forget --keep-last 3 --prune
}

# Loop over the list of repos, running each backup in the background.
for repo in $REPOS; do
  backup_one "$repo" &
done

# Wait for all backups to complete.
wait
2 Likes

Hi @cdhowie ,

Thanks for the tips! I like your solution much better! There’s always more to learn about bash, thanks for teaching me something new!

Sincerely,
jedi453

1 Like