Hi, I have been researching how versioning (VCS, backup and archive) systems might be used with Syncthing and am intrigued by the statement in the Restic documentation that claims support for parallel writes. This is something that I haven’t come across in any other system and gives me hope that it might be possible to use a restic repository in a Syncthing folder shared between multiple devices that may only be intermittently connected.
P.S. As I understand, the Syncthing system (and similar systems like Dropbox) have the following relevant properties:
File writes may not be available to another process on another system immediately. This means that file locks are necessarily impossible.
File changes on one system may not be transfered to another system in the same order that they were made (I think this is configurable in Syncthing, but random order is the default).
If a conflict occurs, the user is presented to deal with it on the level of individual files.
P.P.S. I am looking at using Syncthing rather than other tcp protocols (like sftp, https and minio) because Syncthing already deals with the difficulty of networking nomadic devices without reliable access to IPv6 (NAT traversal stuff).
It’d be interesting to know your use case before trying to explain what may work (and what will not work)
Would you mind describing what you’re trying to do? Make a backup for one machine into a restic repo, which is mirrored to some other machine via syncthing? Do other machines also backup into the same repo?
Of course. My immediate use case is to keep a history of my notes (plain text, org files, images, etc) which are also synchronized with Syncthing. I could limit myself to only writing to the repo with one machine but if i’m going to do that I could use a VCS like Fossil (which uses an sqlite file). It would be interesting if I could write to the repo with any machine at any time without worrying about conflicts. Would the Restic repo file format be compatible with this? It seems like it might be possible since new backups seem to only create new files and never update or delete existing files. However I do see that packfiles are used. If these are updated during backup then it could cause merge conflicts.
Yes, for the backup operation this works just fine. It’s the the forget and prune operations I’m worrying about: They will repack stuff into new files (no files are ever changed, files are just removed and new files added), so this should work with syncthing. I can however construct situations where this fails, e.g. when you run prune on both sides and only then sync around. That may be messy. Although this could be cleaned up by syncing all changes and then running prune again, hm.
No files are ever changed, so there will be no merge conflicts. Whenever data is to be rewritten, restic creates a new file (with a new, almost random file name) in the repo and writes the data to the new file.
If you’re interested in how it works under the cover, the design document is a good starting point.
Just FYI, I havent been able to test this yet because I found out that chunk based compression is not suitable for many small changes such as those that I make to my notes text file. Each change will just store a new copy of the file. It seems delta encoding or solid compression will be much better for space considerations but I haven’t found any existing solutions that support parallel writes. One day I might try to make something that supports parallel writes like restic but uses delta encoding (or maybe solid compression or both). For now, I’ll use Fossil SCM.
