Skipping git-ignored files

Getting Help
1

Hi!

I’m trying to trim down the set of files that restic backs-up from my machine. I have a lot of git repositories cloned on my machine from various current and past projects (around 1600 of them…), and I would like to prevent restic from backing up any files that I’ve set to be ignored by git. These files tend to be intermediate files that I could regenerate if I needed to, so there’s no need to back them up.

Has anyone come up with a workable solution for this kind of problem?

I can probably get git to generate a list of all of the files it’s ignoring for each repository, and then put that list into an exclude file that I feed into restic. However this seems like a somewhat cumbersome way of doing it – if restic knew how to interpret git ignore files then it would be really handy!

2

Sorry but you’ll have to do this with additional tooling. There’s excludes as you already mentioned, there’s also the --files-from* options you can use.

3

You might want to take a look at the discussions in Use .gitignore (or other VCS) for excluding files · Issue #1514 · restic/restic · GitHub

4

Thanks! Your comments pointed me in the right direction: In #2246, which is linked to from #1514 that @MichaelEischer directed me to. Using fd to do the gitignoring results in a working arrangement.

The only complexity I found was that with the --files-from-raw argument results in restic’s cache directory exclusion, so its functionality has to be hacked back in again. Here’s what I ended up with:

# Find the ID of the latest shapshot
parent=$(restic --json snapshots -H $(hostname -s) | jq -r 'max_by(.time) | .id')

restic backup -x --exclude-caches \
        --parent $parent \
        --files-from-raw \
        <(fd -H0 --one-file-system \
	      -t f -t l \
	 	  --ignore-file <(fd -H --one-file-system CACHEDIR.TAG $HOME -x echo {//}/) \
		  --ignore-file ~/.restic-exclude \
		  . $HOME)

I think the -x and --excludes-caches arguments to restic are probably entirely superfluous.

5

… and after playing with that a bit I find I run into the same problem reported in #3451, which will hopefully be resolveable when pull request 3200 is merged :crossed_fingers:

6

One recommendation where you could improve this a bit: You can use the flag --latest 1 to only output the latest snapshot available. Depending on where your repository lives, this could save some time.
Like so:

parent=$(restic --json snapshots --latest 1 -H $(hostname -s) | jq -r 'max_by(.time) | .id')
1 Like