Manage configuration profiles

Hi there,

I’ve been trying restic for a while, as I need to replace my old homemade rsync scripts, and so far it fits all my requirements. Good work!

One thing that came to me straight away is that it involves a lot of typing. I’m using different Azure backends and I had to juggle with different environment variables with different storage accounts.

Another thing I needed is different profiles: I do local backups of the root to a data disk, a backup to another machine locally, then a backup to an azure storage account. Managing different repositories is not the easiest.

For that matter I’ve created a quick python script that can manage profiles. An example is better than a long discussion, and here’s what it can do right now:

  1. A simple configuration file using a Microsoft Azure backend:
[default]
repository = "azure:restic:/"
password-file = "key"

[default.env]
AZURE_ACCOUNT_NAME = "my_storage_account"
AZURE_ACCOUNT_KEY = "my_super_secret_key"

[default.backup]
exclude-file = "excludes"
exclude-caches = true
one-file-system = true
tag = [ "root" ]
source = [ "/", "/var" ]
  1. A more complex configuration file showing profile inheritance and two backup profiles using the same repository:
# Global configuration section
[global]
ionice = false # Linux only
ionice-class = 2
ionice-level = 6
nice = 10 # All unix-like
# when no command is specified when invoking resticprofile
default-command = "snapshots"
# initialize a repository if none exist at location
initialize = false

# a group is a profile that will call all profiles one by one
[groups]
# when starting a backup on profile "full-backup", it will run the "root" and "src" backup profiles
full-backup = [ "root", "src" ]

# Default profile when not specified (-n or --name)
# Please note there's no default inheritance from the 'default' profile (you can use the 'inherit' flag if needed)
[default]
repository = "/backup"
password-file = "key"
initialize = false

[default.env]
TMPDIR= "/tmp"

[no-cache]
inherit = "default"
no-cache = true
initialize = false

# New profile named 'root'
[root]
inherit = "default"
initialize = true

# 'backup' command of profile 'root'
[root.backup]
exclude-file = [ "root-excludes", "excludes" ]
exclude-caches = true
one-file-system = false
tag = [ "test", "dev" ]
source = [ "." ]

# retention policy for profile root
[root.retention]
before-backup = false
after-backup = true
keep-last = 3
keep-hourly = 1
keep-daily = 1
keep-weekly = 1
keep-monthly = 1
keep-yearly = 1
keep-within = "3h"
keep-tag = [ "forever" ]
compact = false
prune = false

# New profile named 'src'
[src]
inherit = "default"
initialize = true

# 'backup' command of profile 'src'
[src.backup]
exclude-file = [ "src-excludes", "excludes" ]
exclude-caches = true
one-file-system = false
tag = [ "test", "dev" ]
source = [ "./src" ]

# retention policy for profile src
[src.retention]
before-backup = false
after-backup = true
keep-within = "30d"
compact = false
prune = true

Here are a few examples how to run restic (using the example configuration file)

See all snapshots:

python resticprofile.py

Backup root & src profiles (using full-backup group)

python resticprofile.py --name "full-backup" backup

If you guys would have any use of it, I’m happy to release a package on PyPi. I just need to tidy up some loose ends (mostly error checking: for now the script expects a clean configuration file)

Fred

1 Like

Seems interesting.

I would like to learn from it.

Hi,

I started it as a simple python script but it quickly became bigger than I expected.

It actually works for my backups right now, but there’s no proper check of the configuration file. So I’m refactoring the script. I’m also adding unit tests in the process, to make it more robust.

It’s available here https://github.com/creativeprojects/resticprofile and I should push it to PyPi shortly. It’s the first time I’m going to make a PyPi package so bare with me :slight_smile:

1 Like

It’s a bit weird that restic doesn’t allow a config file, or even for the most of its flags to be set in the environment (e.g. RESTIC_EXCLUDE_FILE would be nice).

There’s a thread about it on github: https://github.com/restic/restic/issues/16 which I noticed after I started this python script.

Hi,

I’ve made a first release. I’ve been using it for all my servers now. I know there are some other options already, but none of them was ticking all the boxes for me:

  • Easy to install (no need to install a lot of packages: python3 is pre-installed on all the distributions I’m using)
  • Easy to read and write configuration (TOML)
  • Easy to manage multiple repositories (profile inheritance)
  • Capable of accepting file streams from stdin
  • Can run any restic command
  • Can call scripts before and/or after a command
  • Can check repositories before and/or after a backup
  • Can backup to multiple repositories (one after another)
  • Also works with Windows

So far it works for all my needs. You guys can try if it also works for your configuration :slight_smile: (hopefully)

If you’re interested, I made a bash script called rescript that also works with multiple configuration files. There are a couple people using it without any problems AFAIK. You can create new configuration files using rescript config and following the menu.

This is a “just for fun” kind of bash script. I created it initially just for me and to learn about bash scripting. It’s pretty big and uses common tools as wget to update, rsync just for one function that I think I’m the only one using it, getopt for options; basically things preinstalled in almost any gnu/linux distribution. Works on Mac and FreeBSD too (as far as I have tested). It has its own commands and you can use restic commands too calling rescript reponame restic_command --flags options .... It can also use its own flags with restic commands like this: rescript reponame -dlqt -- restic_command --flags options ....

It does look good as well actually :stuck_out_tongue:
I wish there was a place to catalog all these little scripts: After all I don’t think I needed to do yet another one :roll_eyes:

1 Like

Well, I haven’t seen another wrapper like yours in Python. It’s good to have variety. There is this post where there are a couple of scripts. Check out one called restic-runner, that’s one of my favorites.