Exclude syntax confusion

#23

Very good catch, I totally missed that!

0 Likes

#24

Thanks @cdhowie, and @Julian for helping me resolve this issue.

Perhaps as a habit, I put the excluded parameter in quotes (trying to avoid shell expansion of the expression)

I called out my mistake in the post above.

Once again, thank you both.

G

0 Likes

#25

Right, but we’re saying that --exclude="**/footage/**" should work because the shell erases the quotes. However, based on the ps output it looks like the quotes were actually passed to restic.

Were you doing something like --exclude='"**/footage/**"'?

We’re just trying to understand what happened.

0 Likes

#26

I hear ya @cdhowie.

I don’t think the shell erases the quotes. I think the shell will make substitutions if the patten falls into the category of something it knows about for example:

if I call du -sh *c* the shell substitutes the c for something it knows about – I say this because invoking ps on the du process shows:

gene  [gene@tws09 ~]$ ps -ef | grep du
41185  38406  6 11:49 pts/4    00:00:00 du -sh cal cmn

instictively (and wrongly in this case) I put the expression in quotes hoping to prevent the shell from getting involved in the globbing process.

Hope that clarifies.

G

0 Likes

#27

Yes, it should. Compare the output of these commands:

chris@liz:~$ echo --test=foo
--test=foo
chris@liz:~$ echo --test="foo"
--test=foo

Do you get different output?

That’s the strange thing – what you did was actually right. The quotes should be erased by the shell.

The only guess I have is that you are calling restic from a script, and the shebang line in the script invokes a shell that doesn’t erase quotes, and that’s a long shot.

0 Likes

#28

Ha. That’s awesome – I did not know that the shell behaves this way with quotes. Thanks for the schooling. I will watch out for that in the future.
And yes, you are 100% correct as to the script. I’m invoking restic from a python subprocess, which probably trips this behaviour up.

cmd = 'restic -o b2.connections=20 --exclude footage -r b2:projects-{}:{} backup {}'.format(proj_creation_year, project, proj_path)
lcmd = cmd.split(' ')
ps = subprocess.Popen(lcmd, env=restic_env, stdout = subprocess.PIPE, stderr = subprocess.STDOUT)

well – the token says footage now but it used to say "**/footage/**" which is where this whole thing went south for me! :wink:

2 Likes

#29

Ah, yup, there it is! subprocess.Popen() doesn’t invoke a shell, it takes your arguments array and passes them to the target program as-is. This means that quotes and * characters alike are untouched, so using **/footage/** without quotes here is safe.

And, of course, using quotes means they get passed to restic, which will dutifully ignore any paths that start and end with a " character, and have /footage/ somewhere in them. :slight_smile:

Thanks for satisfying my curiosity.


Oh and look, you aren’t the first person to trip up here. I knew this sounded familiar… I helped someone else out with this exact issue!

1 Like

#30

heh - well, at least i’m not alone.

Thank you for your patience in getting to the bottom of this. TIL.

On a side note, I would not be surprised to see the " character in a file name one day… It will be a dark day. :wink:

2 Likes