If I rsync a restic repository, it would be nice being able to lock it first.
Like in
restic lock --mode=exclusive --for-process=$$ …
restic unlock --for-process $$ …
I think this has been proposed before. To avoid dangling locks, it’s better to have restic invoke a subprocess and remove the lock when it terminates, like:
restic lock --mode=exclusive rclone ...
Note that you’d only need a shared lock in this case. There’s no reason to lock the whole repository for rclone since it does not write to the repository.
That would definitively be helpful, but I prefer my variant for those reasons:
-
there might be more than just one command to be enclosed with a lock. For example, I might want to clone the exactly same state to several external storages. Unlock/lock between them allows races.
-
one more example: my variant simplifies parallel execution using what bash offers (like cloning to external USB and over the net in parallel, or to several slow USB drives with fast USB). In your variant, this use case would probably require the definition of one more script to be called by restic.
-
there may be more complex use cases we don’t think about yet.
-
in bash, trap “restic unlock …” EXIT is a very simple and reliable way to avoid dangling locks. However I don’t know (and don’t care) about systems not running bash.
-
somewhere in this forum I read that one should not just always unlock before doing stuff, better look for the reason of the dangling lock. Your variant as it stands contradicts that. It would just ignore bugs in the execution and always unlock. For a solution, see below.
-
right now restic often leaves dangling locks on my systems. Maybe it does not clear them when the process gets killed, I did not yet investigate. Unless restic fixes that, your argument is void. My suspicion: I often use CTRL-C. Typical scenario when creating a new backup set: Define exclude-files and run backup. When I see big and irrelevant things scrolling by, CTRL-C, exclude them, start over. When done, prune. Much better than overflowing the file system by mistake.
If your variant gets implemented:
-
restic lock will not correspond to restic unlock. Better call it restic execute, restic run, restic run_locked or something like that. That would avoid misunderstandings.
-
if the executed code exits<>0: do not unlock and make restic return that code. Or should it unlock even then?
So use sh -c as the command.
restic lock --mode=exclusive sh -c 'rclone ... && rclone ...'
restic lock --mode=exclusive sh -c '
rclone ... &
rclone ... &
wait
'
I don’t understand, so would the trap example you gave. You argue one point then contradict it in the next.
The advice is to avoid running restic unlock to remove an orphaned lock without figuring out what happened. It does not mean that you shouldn’t remove the lock held during execution of a failed command.
No, it means that we should fix the issue where restic leaves dangling locks. A bug existing is not a good reason to choose one feature design over another.
I am absolutely open to different names for the command. The name is not as important to me as the design of the feature’s operation. I like restic run and restic exec.
Restic should forward the exit code and remove the lock.
The trap example should only show that even a bash script can reliably unlock. If the script wants to leave the repo locked, it can explicitly untrap the trap.
Absoluty. I forgot to write that this is all possible but it could result in rather nasty code. Like problems with quoted quotes, arbitrary unicode and whatever. Please keep in mind that a file name can have almost arbitrary unicode. So as soon as a file or directory name is involved, it will IMHO be difficult to make sh -c work worldwide. Maybe I don’t know enough about bash but then most don’t. Maybe other shells are better at that, but since bash is probably one of the more important ones, it is relevant.
Thanks for the clarification. This is what your first post said, I read too much into it.
Right, I just think it’s a bit cleaner for restic to provide this functionality and not require the script author to write the boilerplate. This is a rather annoying thing to get right in Windows batch files, for example.
Right, it can be situational. Perhaps restic should implement both options.
Stupid question Why do you guys need restic to manage locking for your external scripts?
Restic should be concerned with locking for its own operations, but if you’re doing stuff outside of that you should manage your own locking.
That presupposes that the locking mechanism has a documented and stable API.
Oh wait - it it actually documented - I did not yet see that.
https://restic.readthedocs.io/en/latest/100_references.html#locks
This seems a bit much for a simple bash script. But since I use python, that is the way I will go.
The only question I do not see answered in the documentation: are pid and uid the effective or the real values?
well - I overlooked the fact that the lock files are encrypted. Being no expert for encryption, I’d need some pseudo code about how to decrypt/encrypt that.