Some exclude patterns work, some do not. Why?

I am new to restic and like it so far. I am trying to understand why some of these patterns work with the --exclude option and some do not. I run restic from a bash script so I do not have to input the repo path:

#!/usr/bin/bash

IFS="$(printf '\n\t')"

repo="/media/steve/Ext4/restic-repo"

# export RESTIC_PASSWORD=""

if [ ! -d $repo ] 
then
  echo "External drive is not connected."
else
  echo "Starting backup to..." 
  echo "${repo}"
  restic -r ${repo} backup $HOME ${1} \
    --exclude .cache \
    --exclude .local/share/Trash \
    --exclude ".mozilla/firefox/*/storage/default/http**" \
    --exclude ".var/app/*/cache**" \
    --exclude .local/share/flatpak \
    --exclude .local/share/docker \
    --exclude __pycache__ \
    --exclude .config/Slack
fi

exit 0

As you see, some of the patterns are in quotes because I found that necessary to get the exclusions I intended. I started by using patterns that worked in rsync (I realize it’s a different app written in a different language). For example:

--exclude .var/app/*/cache does not work.

I expected this would exclude all the cache directories under .var/app (with ‘*’ being the flatpak app, e.g., com.spotify.Client) and their files and subdirectories (caches for my flatpak installations).

Likewise, these patterns did not work:
.var/app/*/cache**
.var**cache
.var**cache**

Also not working…
.mozilla**default/http** (works in rsync)
.mozilla/firefox/*/storage/default/http**
.mozilla/firefox/*/storage/default/http*

At first, I wondered whether these paths were problematic because they contained some unusual features; e.g., the flatpak app directories all have a pattern of tld.domain.App. And the firefox storage/default directory has subdirectories named as “http[s]+++URL”.

But when I tested this here the multiple “.” or “+” caused no problems.

package main

import (
	"fmt"
	"path/filepath"
)

func main() {
	fmt.Println("On Unix:")
	fmt.Println(filepath.Match("/home/*/foo*", "/home/a.b.c/foobar"))
	fmt.Println(filepath.Match("/home/*/foo*", "/home/a+++c/foobar"))

}

Output:

On Unix:
true <nil>
true <nil>

However, I see that restic does not use pure filepath.Match from go, based upon its use of “**” wildcard and what I see here.

Does anyone have an idea why quoting these patterns seems to work, but all the other options do not?

How come you don’t use the --exclude-file option instead? :slight_smile:

Because I have written a bash script with those instructions as --exclude options. Why write them in a separate file and need to remember if I move that ‘excludes.txt’ file some day? I actually started with --exclude-file but then changed my mind for the reasons above. Anyhow, I don’t think that would have any effect on the matching pattern issue.

After searching this forum a bit more I am pretty sure I found the solution from @MichaelEischer in this thread. Briefly,

If you specify wildcards on the command line without any quoting, then these are expanded by the shell before restic sees the parameters.

The correct syntax would be something like this:

restic backup --exclude "/opt/www/nextcloud/data/*.log"

Note the quotes around the file name.

And my apology to @rawtaz, using an exclude-file would have alleviated the need for quoting the wildcard-containing paths.

4 Likes

You solved it :slight_smile: I’ve been away. Yeah, I just figured that if you use an exclude file you won’t have to dabble with the shell since all the paths in the exclude file are only read by restic and then expanded. Glad it works now, nicely done!

1 Like