Manage configuration profiles

Hi there,

I’ve been trying restic for a while, as I need to replace my old homemade rsync scripts, and so far it fits all my requirements. Good work!

One thing that came to me straight away is that it involves a lot of typing. I’m using different Azure backends and I had to juggle with different environment variables with different storage accounts.

Another thing I needed is different profiles: I do local backups of the root to a data disk, a backup to another machine locally, then a backup to an azure storage account. Managing different repositories is not the easiest.

For that matter I’ve created a quick python script that can manage profiles. An example is better than a long discussion, and here’s what it can do right now:

  1. A simple configuration file using a Microsoft Azure backend:
[default]
repository = "azure:restic:/"
password-file = "key"

[default.env]
AZURE_ACCOUNT_NAME = "my_storage_account"
AZURE_ACCOUNT_KEY = "my_super_secret_key"

[default.backup]
exclude-file = "excludes"
exclude-caches = true
one-file-system = true
tag = [ "root" ]
source = [ "/", "/var" ]
  1. A more complex configuration file showing profile inheritance and two backup profiles using the same repository:
# Global configuration section
[global]
ionice = false # Linux only
ionice-class = 2
ionice-level = 6
nice = 10 # All unix-like
# when no command is specified when invoking resticprofile
default-command = "snapshots"
# initialize a repository if none exist at location
initialize = false

# a group is a profile that will call all profiles one by one
[groups]
# when starting a backup on profile "full-backup", it will run the "root" and "src" backup profiles
full-backup = [ "root", "src" ]

# Default profile when not specified (-n or --name)
# Please note there's no default inheritance from the 'default' profile (you can use the 'inherit' flag if needed)
[default]
repository = "/backup"
password-file = "key"
initialize = false

[default.env]
TMPDIR= "/tmp"

[no-cache]
inherit = "default"
no-cache = true
initialize = false

# New profile named 'root'
[root]
inherit = "default"
initialize = true

# 'backup' command of profile 'root'
[root.backup]
exclude-file = [ "root-excludes", "excludes" ]
exclude-caches = true
one-file-system = false
tag = [ "test", "dev" ]
source = [ "." ]

# retention policy for profile root
[root.retention]
before-backup = false
after-backup = true
keep-last = 3
keep-hourly = 1
keep-daily = 1
keep-weekly = 1
keep-monthly = 1
keep-yearly = 1
keep-within = "3h"
keep-tag = [ "forever" ]
compact = false
prune = false

# New profile named 'src'
[src]
inherit = "default"
initialize = true

# 'backup' command of profile 'src'
[src.backup]
exclude-file = [ "src-excludes", "excludes" ]
exclude-caches = true
one-file-system = false
tag = [ "test", "dev" ]
source = [ "./src" ]

# retention policy for profile src
[src.retention]
before-backup = false
after-backup = true
keep-within = "30d"
compact = false
prune = true

Here are a few examples how to run restic (using the example configuration file)

See all snapshots:

python resticprofile.py

Backup root & src profiles (using full-backup group)

python resticprofile.py --name "full-backup" backup

If you guys would have any use of it, I’m happy to release a package on PyPi. I just need to tidy up some loose ends (mostly error checking: for now the script expects a clean configuration file)

Fred

2 Likes

Seems interesting.

I would like to learn from it.

1 Like

Hi,

I started it as a simple python script but it quickly became bigger than I expected.

It actually works for my backups right now, but there’s no proper check of the configuration file. So I’m refactoring the script. I’m also adding unit tests in the process, to make it more robust.

It’s available here https://github.com/creativeprojects/resticprofile and I should push it to PyPi shortly. It’s the first time I’m going to make a PyPi package so bare with me :slight_smile:

2 Likes

It’s a bit weird that restic doesn’t allow a config file, or even for the most of its flags to be set in the environment (e.g. RESTIC_EXCLUDE_FILE would be nice).

1 Like

There’s a thread about it on github: https://github.com/restic/restic/issues/16 which I noticed after I started this python script.

1 Like

Hi,

I’ve made a first release. I’ve been using it for all my servers now. I know there are some other options already, but none of them was ticking all the boxes for me:

  • Easy to install (no need to install a lot of packages: python3 is pre-installed on all the distributions I’m using)
  • Easy to read and write configuration (TOML)
  • Easy to manage multiple repositories (profile inheritance)
  • Capable of accepting file streams from stdin
  • Can run any restic command
  • Can call scripts before and/or after a command
  • Can check repositories before and/or after a backup
  • Can backup to multiple repositories (one after another)
  • Also works with Windows

So far it works for all my needs. You guys can try if it also works for your configuration :slight_smile: (hopefully)

1 Like

If you’re interested, I made a bash script called rescript that also works with multiple configuration files. There are a couple people using it without any problems AFAIK. You can create new configuration files using rescript config and following the menu.

This is a “just for fun” kind of bash script. I created it initially just for me and to learn about bash scripting. It’s pretty big and uses common tools as wget to update, rsync just for one function that I think I’m the only one using it, getopt for options; basically things preinstalled in almost any gnu/linux distribution. Works on Mac and FreeBSD too (as far as I have tested). It has its own commands and you can use restic commands too calling rescript reponame restic_command --flags options .... It can also use its own flags with restic commands like this: rescript reponame -dlqt -- restic_command --flags options ....

1 Like

It does look good as well actually :stuck_out_tongue:
I wish there was a place to catalog all these little scripts: After all I don’t think I needed to do yet another one :roll_eyes:

2 Likes

Well, I haven’t seen another wrapper like yours in Python. It’s good to have variety. There is this post where there are a couple of scripts. Check out one called restic-runner, that’s one of my favorites.

1 Like

Hello I am testing resticprofile. Where can I find an example of what line or command to run in Windows Task Scheduler?

I just call the file resticprofile.exe? With the trigger at the time of the backup?

Perhaps at
https://creativeprojects.github.io/resticprofile/

1 Like

The entry in the task scheduler is managed for you.

The only thing you need to schedule all your profiles (after writing your configuration) is typing the command

resticprofile schedule --all
1 Like

OK I have this now on Power Shell:

PS C:\restic> resticprofile schedule --all
2023/06/26 20:06:02 using configuration file: profiles.toml

Analyzing backup schedule 1/1
=================================
  Original form: 12:30
Normalized form: *-*-* 12:30:00
    Next elapse: Tue Jun 27 12:30:00 -03 2023
       (in UTC): Tue Jun 27 15:30:00 UTC 2023
       From now: 16h23m58s left

2023/06/26 20:06:02 scheduled job default/backup created

If I understand well now I also need to add a entry on the Task Scheduler to run “resticprofile.exe” at 12:30 right?

Yes, I read the docs but there is no sample for Windows.

Edited: Schedule Examples :: resticprofile

Found here!

@creativeprojects Is there a way to minimize the DOS window that says resticprofile while the scheduled backup is running?

You can try to change the flag schedule-permission. I’m no longer using Windows and I can’t remember which one makes the console window hidden.

More documentation here

1 Like

No problem, I’m using something like this:

Set WshShell = CreateObject("WScript.Shell")
WshShell.Run chr(34) & "mybackup.bat" & Chr(34), 0
Set WshShell = Nothing