Very good catch, I totally missed that!
Thanks @cdhowie, and @Julian for helping me resolve this issue.
Perhaps as a habit, I put the excluded parameter in quotes (trying to avoid shell expansion of the expression)
I called out my mistake in the post above.
Once again, thank you both.
G
Right, but we’re saying that --exclude="**/footage/**" should work because the shell erases the quotes. However, based on the ps output it looks like the quotes were actually passed to restic.
Were you doing something like --exclude='"**/footage/**"'?
We’re just trying to understand what happened.
I hear ya @cdhowie.
I don’t think the shell erases the quotes. I think the shell will make substitutions if the patten falls into the category of something it knows about for example:
if I call du -sh *c* the shell substitutes the c for something it knows about – I say this because invoking ps on the du process shows:
gene [gene@tws09 ~]$ ps -ef | grep du
41185 38406 6 11:49 pts/4 00:00:00 du -sh cal cmn
instictively (and wrongly in this case) I put the expression in quotes hoping to prevent the shell from getting involved in the globbing process.
Hope that clarifies.
G
Yes, it should. Compare the output of these commands:
chris@liz:~$ echo --test=foo
--test=foo
chris@liz:~$ echo --test="foo"
--test=foo
Do you get different output?
That’s the strange thing – what you did was actually right. The quotes should be erased by the shell.
The only guess I have is that you are calling restic from a script, and the shebang line in the script invokes a shell that doesn’t erase quotes, and that’s a long shot.
Ha. That’s awesome – I did not know that the shell behaves this way with quotes. Thanks for the schooling. I will watch out for that in the future.
And yes, you are 100% correct as to the script. I’m invoking restic from a python subprocess, which probably trips this behaviour up.
cmd = 'restic -o b2.connections=20 --exclude footage -r b2:projects-{}:{} backup {}'.format(proj_creation_year, project, proj_path)
lcmd = cmd.split(' ')
ps = subprocess.Popen(lcmd, env=restic_env, stdout = subprocess.PIPE, stderr = subprocess.STDOUT)
well – the token says footage now but it used to say "**/footage/**" which is where this whole thing went south for me! 
2 Likes
Ah, yup, there it is! subprocess.Popen() doesn’t invoke a shell, it takes your arguments array and passes them to the target program as-is. This means that quotes and * characters alike are untouched, so using **/footage/** without quotes here is safe.
And, of course, using quotes means they get passed to restic, which will dutifully ignore any paths that start and end with a " character, and have /footage/ somewhere in them. 
Thanks for satisfying my curiosity.
Oh and look, you aren’t the first person to trip up here. I knew this sounded familiar… I helped someone else out with this exact issue!
1 Like
heh - well, at least i’m not alone.
Thank you for your patience in getting to the bottom of this. TIL.
On a side note, I would not be surprised to see the " character in a file name one day… It will be a dark day.
2 Likes