Getting started with resticprofile

Hi,

Here’s a copy of a very simple Getting Started with restic and resticprofile.

The new syntax completion of the resticprofile configuration file makes it very easy to create a configuration without remembering all the restic flags that you need.

Prerequisite

resticprofile is one of many automation tools for restic, also called a wrapper.

In a nutshell, resticprofile provides a configuration file and a runner that will generate all the necessary calls to restic.

Unless you’re using the resticprofile Docker image, you need to have restic installed on your machine.

Choose your favourite format

resticprofile configuration file can be written in:

  • TOML : configuration file with extension .toml or .conf
  • YAML : configuration file with extension .yaml
  • JSON : configuration file with extension .json
  • HCL: configuration file with extension .hcl

We recommend using either TOML or YAML.

JSON is fine for auto generated configurations but is not the easiest format for a human to read and write.

HCL can be interesting if you already use a tool from the Hashicorp stack otherwise that’s yet another format to learn.

Configure your text editor

We’re going to show you how to get documentation and auto-completion for the resticprofile configuration using Visual Studio Code.

You can use any other editor that recognise the JSON schema. The same JSON schema can be used for JSON, TOML and YAML file formats.

TOML

In Visual Studio Code, you need to install an extension that supports completion and syntax validation using a JSON schema.

For example you can install the Even Better TOML extension:

TOML extension

YAML

Same for YAML: in Visual Studio Code you need to install an extension like the one provided by Red Hat to understand the shape of your configuration file.

YAML extension

Write your first configuration file

The configuration file is essentially a list of profiles containing a list of commands and flags.

A profile defines:

  • a restic repository (flag repository)
  • how to access your restic repository (like password-file)
  • a list of commands (like backup):
    • which files to include in a backup (flag source)
    • when to backup them (we’ll see that later)
    • any other restic flags you would add to the command line (like verbose)

In this example, we’re going to call our profile default. You don’t need to specify the profile name on the command line when using this name (default).

Create a file named profiles with the extension of your choice (.toml, .yaml, .hcl or .json)

TOML

#:schema https://creativeprojects.github.io/resticprofile/jsonschema/config-1.json
#
# indentation is not needed but it makes it easier to read ;)
#
version = "1"

[default]
  repository = "local:/backup"
  password-file = "password.txt"

  [default.backup]
    verbose = true
    source = [ "/home" ]

YAML

# yaml-language-server: $schema=https://creativeprojects.github.io/resticprofile/jsonschema/config-1.json

version: "1"

default:
  repository: "local:/backup"
  password-file: "password.txt"

  backup:
    verbose: true
    source:
      - "/home"

Generate a secure password

resticprofile has a handy command that can generate a cryptographically secure password file for you:

resticprofile generate --random-key > password.txt

Initialize your repository

Now that you have a password file, you can initialize your restic repository:

resticprofile init

Here’s an example of the result:

2023/03/25 15:46:48 using configuration file: profiles.yaml
2023/03/25 15:46:48 profile 'default': starting 'init'
created restic repository e21ab75046 at local:/backup

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.
2023/03/25 15:46:51 profile 'default': finished 'init'

Test your first backup

Before going live, you can dry run your first backup by using the resticprofile --dry-run flag.

resticprofile --dry-run backup

And here’s the result:

2023/03/25 15:49:51 using configuration file: profiles.yaml
2023/03/25 15:49:51 profile 'default': starting 'backup'
2023/03/25 15:49:51 dry-run: /usr/local/bin/restic backup --password-file password.txt --repo local:/backup --verbose /home
2023/03/25 15:49:51 profile 'default': finished 'backup'

As you can see, resticprofile converted your backup profile into this command line:

/usr/local/bin/restic backup --password-file password.txt --repo local:/backup --verbose /home

Flags

Let’s stop a moment and analyse the command line: we passed the flag --dry-run before the command backup: it means the flag is used by resticprofile.

Let’s try again with the flag after the command:

resticprofile backup --dry-run

And the result is rather different:

2023/03/25 15:50:02 using configuration file: profiles.yaml
2023/03/25 15:50:02 profile 'default': starting 'backup'
open repository
repository e22aa770 opened (version 2, compression level auto)
created new cache in /Users/CP/Library/Caches/restic
lock repository
no parent snapshot found, will read all files
load index files
start scan on [/home]
start backup on [/home]
scan finished in 0.202s: 0 files, 0 B

Files:           0 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Data Blobs:      0 new
Tree Blobs:      1 new
Would add to the repository: 346 B (292 B stored)

processed 0 files, 0 B in 0:00
2023/03/25 15:50:03 profile 'default': finished 'backup'

If you add flags after the command, the flags will be sent to restic instead. As you can see, restic simulated a backup of your /home folder.

Schedule

Let’s imagine you want to backup your files every day during your lunch break.

Add a line in your configuration (in the default → backup section) with an option called schedule and a value of 12:30. Your configuration should now look like:

TOML

#:schema https://creativeprojects.github.io/resticprofile/jsonschema/config-1.json
#
# indentation is not needed but it makes it easier to read ;)
#
version = "1"

[default]
  repository = "local:/backup"
  password-file = "password.txt"

  [default.backup]
    verbose = true
    source = [ "/home" ]
    schedule = "12:30"

YAML

# yaml-language-server: $schema=https://creativeprojects.github.io/resticprofile/jsonschema/config-1.json

version: "1"

default:
  repository: "local:/backup"
  password-file: "password.txt"

  backup:
    verbose: true
    source:
      - "/home"
    schedule: "12:30"

resticprofile can schedule work on macOS, Windows, most Unixes and Linux distributions: it is simply adding an entry in the default scheduler of your platform.

To schedule the backup of the default profile, simply type the command:

resticprofile schedule

Now your backup will run every day at 12:30. As simple as that!

Inline help

You can get help for any command at any time.

The help is available for both the resticprofile internal commands and all the restic commands.

resticprofile help

To get the help on a resticprofile command, simply use the help command or command -h:

Example:

$ resticprofile help generate

The "generate" command is used to create various resources and print them to stdout

Usage:
  resticprofile [resticprofile flags] generate [command specific flags]

Flags:
  --bash-completion                               generate a shell completion script for bash
  --config-reference [--version 0.15] [template]  generate a config file reference from a go template (defaults to the built-in markdown template when omitted)
  --json-schema [--version 0.15] [v1|v2]          generate a JSON schema that validates resticprofile configuration files in YAML or JSON format
  --random-key [size]                             generate a cryptographically secure random key to use as a restic keyfile (size defaults to 1024 when omitted)
  --zsh-completion                                generate a shell completion script for zsh

restic help

This is the same syntax to display the help from a restic command:

$ resticprofile init -h

The "init" command initializes a new repository.

EXIT STATUS
===========

Exit status is 0 if the command was successful, and non-zero if there was any error.

Usage:
  resticprofile [resticprofile flags] [profile name.]init [flags]

Flags:
[...]

Now have fun configuring your backups!

4 Likes

It seems useful! Thanks for that!

2 Likes

I setup a profile with multiple groups and inheritance. It works great. Toml seems to be pleasant to work with.

Resticprofile is a fantastic tool!

Wish I had known sooner …


PS

In your Toml examples the backslash is often escaped.

For example

restic-binary = "c:\\ProgramData\\chocolatey\\bin\\restic.exe"

But if you use single quotes in Toml no escaping is performed, so this would also work, right?

restic-binary = 'c:\ProgramData\chocolatey\bin\restic.exe"
1 Like

Yes I believe it would also work. Actually, it would even handle / instead of \ (I know it’s a funny one)

restic-binary = "c:/ProgramData/chocolatey/bin/restic.exe"
1 Like

Thanks.

Resticprofile is so good, I renamed it to restic.exe (or restic on Linux) actually :smile:

It can do everything basically and replaced all my batch files completely.

There is only one thing that I haven’t figured out yet (on WIndows)

On Windows my backup paths are relative (to avoid having the /C/ prefix in the backup), and because of that I have to run restic or resticprofile from the root folder (C:)

So my backup command looks like this currently

C:\Users\> pushd C:\ & resticprofile -v all.backup -v & popd

How could I move the change directory part to profiles.toml?

I tried

  run-before = 'pushd C:\'
  run-after = 'popd'
  run-after-fail = 'popd'

But that didn’t work unfortunately.

1 Like

No that’s right you can’t do that: each run-* option is running in its own shell.

I never had any need for it yet, but I suppose I can add a root option to define the root of relative paths.

1 Like

Awesome! That would be truly great!


And here maybe something not so essential. But I was wondering …

My backup command looks like that currently

resticprofile -v all.backup -v | tee restic_log.txt | sed "/^unchanged/d"

Obviously it would look nicer like that

resticprofile -v all.backup -v

with “pipe options” in the configuration file. Not sure if this is something easy to implement?

Just brainstorming.

Anyway, thanks again for resticprofile! Awesome project. So many options. An engineering marvel (oops, restic has entered the chat … )

1 Like

It’s already been asked :slight_smile: Add 'stdout-command' to 'dump' command · Issue #153 · creativeprojects/resticprofile · GitHub

1 Like

It seems there is a pull request for that already.

:slight_smile:

1 Like

How to correctly type group-by: '' in yaml?

1 Like

Sorry, I realise this is a bug introduced in version 0.21.0
You should be able to use resticprofile version 0.20.0 with a flag group-by: ''

2 Likes

Hi,

resticprofile looks cool! And so does restic! I’m a restic newbie trying to get started properly.

Does the docker image also support schedules? If so how would running it look like?

I would except it to have to run something like docker -d ... resticprofile schedule
instead of docker --rm ... resticprofile schedule
but the docker process just exists after printing something like this

2023/09/10 14:04:50 scheduled job default/backup created

I imagine that it does not work because we need it to execute the crond process, and not the resticprofile in the docker. How to do that?

I’m also confused about how to setup rclone, but I’ll take that as step two… First things first…


ps: I’m using this official docker image: Docker :: resticprofile

1 Like

No actually you can’t use the automatic scheduling from the configuration if you’re using the docker image. To be able to work, you need to install either crond or systemd and start the container doing nothing (but keep it running).

Now that I think of it, I could make a special option where resticprofile tries to run itself inside docker. It would have to be installed on the host machine (no need for restic on the host though) and it would setup a command line using docker. I suppose that could work :+1:

2 Likes

hi i am windows user, would like to have more example .yaml files to implement multiple backup profiles.
is it possible to have multiple backups inside the default profiles.yaml?
is it possible to have like below to have all 3 inside a profiles.yaml or i need to create 3 seperate profiles?

example:

  1. backup-set-1 (local) = source-1 to repo1 with includes and excludes backup settings
  2. backup-set-2 (rclone to onedrive) = source-2 to repo2 with includes and excludes backup settings
  3. backup-set-3 (rclone to gdrive) = source-3 to repo4 with includes and excludes backup settings
1 Like

You would definitely need 3 profiles.
As a rule of thumb you can imagine a profile being one restic backup command (eventually associated with a check, forget, or other side commands)

But all the common flags can be added to a base profile, and your 3 profiles can inherit from the base profile.
With this setup you don’t need to copy the common options on each profile :+1:

There’s an example of a configuration with inheritance for Windows in the documentation.

1 Like

noted. regarding the inheritance approach. the first time i studied it just too complicated.
now i tried again, seems like i can understand slightly more and get a sense out of it.

another question, i have “myinclusion-rules.txt” & “myexclusion-rules.txt” where i use the flag of " --filter-from" when using plain restic. how am i able to use this in resticprofile without maintaining two sets, 1 for when using with restic and another 1 when using with autoprofile?

1 Like

Ideally, you’re only using resticprofile once your profile is setup properly.
If you still need to use restic directly, it means I missed something :laughing:

Why do you need two different set of exclusion rules?

1 Like

not exactly 2 sets of rules. anyway, after spending more time playing and testing, resticprofile is indeed very powerful and simple to use after setup it up correctly. i use excel spreadsheet to help me remember and dynamic change to all detail of each of my profiles.

in fact, i am able to do multi-repo and multi-sub backup profile in one single .yaml or using the includes: to combine multiple .yaml files. AWESOME!!!

resticprofile also can easily schedule all my sub-profile in one single commands to windows Task Scheduler. super time saver.

also, i can use files-from and exclude-file to read my inclusion-rules and exclusion-rules files as resticprofile can pass the flags to restic

i have one issue that i need to run “chcp 65001” in Windows command prompt/.bat file or run “export LANG=en_US.UTF-8” in Linux to be able to backup my Chinese characters files. reading from the post above, it seems the "run-*’ commands will be running in another process. How can i do the command run in the same process so that restic can read Chinese characters?

also, checking GitHub, the latest release was so long ago in 2021, will there be update in future for this amazing software?

How can i do the command run in the same process so that restic can read Chinese characters?

For Linux there’s a new way of transferring environment variables between run commands. It won’t help you on Windows though :slightly_frowning_face:

also, checking GitHub, the latest release was so long ago in 2021, will there be update in future for this amazing software?

Where are you looking at? the latest version is 3 months old (I’m planning a new version soon)

seems to be working for backup up Chinese charater files now.

regarding the version, seems like i misread the date.

thanks for the reply

1 Like