No, there’s no such option currently. What actual problem are you trying to solve by wanting the paths to be the same even though you back up from different systems? The decuplication in restic works the same regardless of what paths are recorded for a snapshot, as restic looks at blocks when backing up.
I’d want to have a single dedicated machine for backup with all backups done by pulling. In other words no restic binary on the source machine, no cron jobs on source machine.
I guess I’m not the first to desire the kind of setup.
To avoid this path artifact I just need a way to access a remote source but afaik there is no equivalent sftp for source like there is for target.
So that’s not a problem if I use sshfs or (other method at involves a local mount) but then I end up with this path artifact (of the mount) in the repo snapshot which is not great as that is setup dependent would be confusing and could make restoring from another path problematic.
Note: There is already a way to rewrite/replace the hostname
-H, --host hostname set the hostname for the snapshot manually. To prevent an expensive rescan use the "parent" flag
That I would need to use in the remote mount case since the host will be the backup server not the host of the source (see my snapshot output).
So looks like -P isn’t taken and that could be used for this path rewrite
maybe like so?
-P /new/base/path#/path/to/match
so in my case -P /#/mnt/238/gate/
would rewrite all /mnt/238/gate/opt to /opt
Having this option doesn’t involve restic knowing where/how the source lives but does allow one (for whatever reasons) to keep a consistent base path in the repo no matter how it is backed up. So maybe that is within the scope of this project?
This is what I don’t see the problem in. How is it confusing, considering that you already have your snapshots separated by the hostname?
And how is restoring the files a problem - you can just restore it to a temporary folder and then move the folders wherever you want, e.g. instead of getting /my-restore-point/opt/ you get /my-restore-point/mnt/238/gate/opt/ and can simply move that opt/ folder to / or where you want it, after restoring. It’s just one extra mv command (this is the main reason I asked what the actual or concrete problem is/was).
On a related note, if you’re using a separate system to restore as well, then presumably you already want the path to be /mnt/238/gate/opt instead of /opt, no? Or are you not restoring using this separate system, only backing up?
There are already discussions about this feature, but it’s not implemented merged yet. I think it’s rare to see actual concrete use cases/needs for it, if I may say so. But feel free to try that PR, it sounds like it would do what you want it to
thanks @rawtaz for pointing out this PR. Looks like it’s indeed waiting to be merged so the answer to my question is yes, soon. In the meantime I’ll build with the pr and try it out.
Please do not assume that things will be merged just because they’re requested. The answer to your question is that restic does not have a way to rewrite paths. That PR will undergo review and conserations like every other piece of code that made or didn’t make it into restic. For good reasons.
Can you please answer my questions earlier? In what practical way is it a problem that you have to write one additional mv command after restoring your files (if this is even the case, considering that on that “external” backup server the paths you restore could easily match the ones you backed up)? And how is it confusing when you clearly see which host the /opt and similar paths belong to? In other words, how is this an actual practical concrete problem rather than just a cosmetic annoyance?
I cloned the pr repo, merged in all your commits since april and built restic. Then I tried it out with a bash script i’ve been building (which will eventually work into a nodejs/javascript app), and it works find for me.
I don’t really have anything to add in terms of my use case over the several comments in the issue. I agree with them.
I think the operative word used was “you” i.e. me. Yes it’s all clear if I did the backup and if I do the the restoring what happened. If I maintain a database or a yaml file (like I am using in the script) then a program can know where it really came from/goes but the snap IMO needs to reflect the actual path on the actual machine which is not an issue usless one is trying to setup up a pull backup server.
Also say I write some code to grab the json from an existing repo of snaps a program can then recreate exactly where that snap came. Otherwise without an external database associated with that repo there would be no way.
I have plans for restic. I’ve tried out several potential backup backends for a backup pull server (web) application and restic is clearly the best and I like that it is written in GO. I already have a start on a nodejs wrapper api for restic and I can now move the logic of my bash script into this wrapper
At this point if you/devs don’t/can’t merge this pr I can continue to pull recent restic commits and rebuild with this pr (the beauty of open source). I’ll just include my build as part of my app and make a donation to this project when my app is ready for prime time. I appreciate all the work and the fact that the project is active (which isn’t the case for serveral others I considered)
Well, it looks like it didn’t work as expected. “set-path” set the path property to /opt per the snapshot report (and json) but the actual snapshot of /opt was written within /mnt/238/gate.
so looks like either set-path is not working as intended or it was never intended/able to rewrite the path only to set the path property key of the snapshot.
I agree this would be nice. My use case is that I’m backing up an LVM snapshot, and I’d rather strip the /mnt/snapshot prefix from the backups because it’s much cleaner that way. And for obvious reasons I don’t want to restore to the snapshot directory. Sure, I can specify a different path to restore and mv it afterwards, and that would be technically instant in my case as it’s on the same filesystem. But consider the case where someone is backing up multiple filesystems in one backup, then the move would add an unnecessarily slow copy/delete process.
Also, if you happen to be backing up the same machine from multiple different paths (in my case, I’m trying to import some old tar backups into restic but I don’t want that to clash with my automated daily backup cronjob, so I backup from a different path). If you restore that manually it may not be a big deal to move it into the right place, but if you want to write any sort of automation with it, it’s much nicer if the restore software doesn’t have to deal with figuring out where the real root of the backup starts.
I feel like @rawtaz’s tone in this thread was uncalled for, considering OP seemed to be respectful and didn’t act entitled. They weren’t demanding that it must be implemented. As they said the beauty of open source is that it’s possible to implement it yourself in a fork, but it’s always nicer if something can be included in upstream and more people can benefit with it without an unnecessary fork having to be maintained, as long as it’s reasonable and doesn’t cause too much burden on the upstream maintainers.
Unless I’m misunderstanding you, that’s a pretty good explanation of a use case that makes sense.
I’m curious if this is an actually common use case though, but fair enough!
I think there’s been a misunderstanding. But if I did come across in a bad way then my apologies for that. The OP suggested the PR was sitting there waiting to be merged, while in reality its next step would be a review (which is quite different), that’s why I wrote the comment about not assuming a merge or similar.
I also did re-read everything I wrote and honestly I don’t see anything hostile. It was mostly stating facts, some helpful pointer to a PR and of course a bunch of repeated questioning (which IMO is due to OP not actually answering the questions, that’s why I kept asking). For my educational purposes, which part(s) of what I wrote did you think had a “tone” to them that was not just neutral facts/questions? Thanks.
I wholeheartedly agree with this. The more we can keep software projects together, the better. Totally with you on that!
To comment on stripping path components/prefix, IMO this is something that has been asked enough times that it warrants serious consideration. Whatever the use case is, it’s apparently something that people would benefit from, regardless of whether it’s effectively just for cosmetics or if it’s to solve an actual real/practical problem.
Hey, I ended up in a similar situation when migrating from a different backup solution (I restored the backups I wanted to keep on an external drive, backuped them to restic from there so now the migrated ones have paths like /run/media/user/whatever and the new ones /home/user).
I might be wrong here, but my impression is that restic forget groups snapshots by path, so that if I have a rule to keep only the last 2 snapshots it will keep the last two for /home/userand the last two for /run/media/user/whatever, which is not what I am trying to achieve… there might be some way to work around it by using different rules on different snapshots selected by tag, but IMHO it would be nicer to be able to correct the path for the imported snapshots and manage everything seamlessly
@doscott Thank you very much for your suggestion, it helped in my case
I was thinking that there might be a similar use case where the host is not the same (e.g. migrating from one machine to the other while changing username, so that both the the host and the path would be different). I guess in that case you can use --group-by tag instead, after ensuring that all relevant snapshots have the same tags, or create the new snapshots with --host new-hostname.
I’d say renaming the paths is not really needed, but would still be a nice-to-have feature if it’s not too hard to implement
Anyway, thank you very much for the hint, my individual problem is solved!
Before the backup I create snapshots of the volumes to be backed up with LVM, then mount them and want to back them up but under the original name. To me this seems a perfectly valid scenario. What is wrong with implementing it?
Nothing, the short answer why it isn’t implemented is time. There have so far been several suggestions how the corresponding CLI options could look like, but not enough time on the side of the maintainers to decide how it should look like.
I use Linux mount namespaces to work around this. My restic wrapper takes ZFS snapshots of all pools, does a unshare(CLONE_NEWNS), sets private mounts, sets up the sandbox with snapshot mounts, bind mounts what needs to be read-write like caches, tmpdir, log dir, does the pivot_root dance to create a mirror view of the real filesystem and runs restic as per usual. This way paths show as they really are in the sandbox and backups are atomic and don’t run off the live changing system. A namespace alone isn’t enough as a bind mounted path in child ns disappears if the source file/dir is unlinked in the parent. E.g., if you bind-mount a file in child ns, edit and save with vim in parent, it will disappear from the child ns as vim unlinks and renames on save. By running off snapshots which are read-only also in the parent ns the sandbox is shielded from such behaviour.
I bind mount /dev and /run into the sandbox rw to make journal logging work seamlessly and to be able to notify systemd service manager as the wrapper is of type notify. But as it handles alll sandboxing itself, it runs just as well without systemd and I don’t need to rely on PrivateMounts=yes or other sd sandboxing.
I don’t unshare user as I don’t want to deal with uid/gid mappings and want to ensure backed up files have correct permissions. This means running as root which is fine by me as the system is shielded by the sandbox. Snapshots are destroyed on exit and all mounts torn down automatically by kernel leaving no trace of anything behind.
