I like to keep a local repository for systems being backed up to recover from human error (“oops, didn’t mean to delete that”) but keep a remote one for catastrophic failures, including hardware failures and attacks by malicious third parties. In the case of ransomware in particular, it’s vital to have an off-system append-only copy that the attacker cannot encrypt.
The problem is that there doesn’t seem to be a good way to implement this currently. There’s basically two options:
Off-system, run an append-only REST server as well as a read-only mechanism (HTTP) for reading the repository. Back up to the REST server, then synchronize to the local repository from that.
This requires running two services; we need a secondary server to sync the changes back since nothing but restic appears to talk the client half of the restic REST server protocol.
Requires sending the data out and then fetching it back; inefficient, and also possibly needlessly expensive when outbound traffic from the off-system server is metered.
Back up locally, then copy all new data to the off-system append-only REST server.
There doesn’t appear to be any existing tool that can perform this copying.
Use some other rclone-supported server that has an append-only storage mode.
I’m not sure that any exist.
Am I missing something here? Is this possible today without resorting to the terrible first option?
What do you mean by append-only exactly, at the file-system level? If a file is changed locally (and it is a legitimate change), you would want that change copied to the “append-only” remote, right?
The filesystem ACLs are not relevant, and so I’m not sure what you mean by “at the file-system level.” This is a service-level constraint; see the --append-only flag of the REST server for a description of this mode.
I can’t really figure out what this means, either. Files in a restic repository are never changed. So no, I would not want any changes copied because a change is either a bug or malicious.
I would only want new files to be copied, and I would want the remote service to enforce this.
We already have a service that enforces this (rest-server --append-only) but we do not have a client (to my knowledge) that can use this service as a synchronization destination.
Or the write has not finished yet (or was interrupted) and yes, I think this does depend on the file system, depending on how it coordinates readers and writers.
Not asking about ACLs, just asking how you define an append. An append is a kind of modification – so if a file is appended to, is that a valid change?
If there’s going to be a service that enforces “append-only” we need to be specific about what “append” means.
If file names are a hash of their raw content (I believe this is the case?) the append-only service can trivially detect and simply reject incomplete uploads.
I assumed this term was fairly well-understood as the mode implemented by the REST server?
The goal is to allow new content to be added but disallow the removal or corruption of existing content. Any definition of “append only” that accomplishes that goal is fine with me.
You’re talking about a failed write so that restic retries the upload/save operation? When a file is written to a backend within restic, the data is already complete (so we know the SHA2 hash of its contents) and sits waiting in a temp file. The local backend will then open the file in the repo (with O_CREATE and O_EXCL), copy the content from the temp file over, then flush the file and close it. If anything goes wrong, the file is removed and restic tries again.
So, after the file is closed successfully, restic won’t touch it again.
I thought of a possible solution. On the system being backed up, serve the local repository somehow, such as over HTTP (with IP whitelist, firewall, VPN, and/or authentication to limit access) and then use rclone copy --ignore-existing on the off-site system to pull from the local system, but not overwrite any files that already exist. (Obviously other protocols could also be used.)
This way, instead of pushing data to the off-site system with a service that must be configured not to accept deletions or modifications, the off-site system pulls just the new files.
Obviously, this only works if there are no concurrent backups. Otherwise, it may be necessary to run restic check following a copy to ensure everything is okay.
Edit: There may be a much better rclone option:
--immutable Do not modify files. Fail if existing files have been modified.