restic backup needs to be able to create new objects in S3
S3 PutObject would allow to overwrite existing documents which is not secure
There are two possible solutions:
don’t save S3 credentials on the machine that is running restic, but on a different one. The easiest solution is, as @martinleben proposed, running rclone --apend-only on this separate machine)
activate object versioning in S3. Then S3 would save an additional version of an object and you can always recover an old version.
About prune (and forget, etc.):
When pruning, restic will also need to remove objects. This is done by the RemoveObject API call.
without object versioning, RemoveObject just works as intended.
with object versioning enabled, RemoveObject by default only adds a new marker “removed”. To really remove Objects, you hence need a postprocessing (which you should be very careful about how to implement in a secure way!). The other option is to give RemoveObject the actual version of the object. This is however not supported by restic.
So in short: IMO you should take a look at 1), i.e. the solution without object versioning, but with an additional host.
Thank you all. We are used to having a pretty secure way of doing it from the borg feature set where one can limit to writing new files but not allow it to overwrite files and limit pruning of old backups to another (isolated) server.
Thank you. Can you explain this in more detail?
My understanding is the following - I might be wrong.
run Restic Rest Server with append-only mode
use rclone on a more secure server to copy the Rest Server files to S3
third server which performs the pruning of old data on 1. and is not limited by append-only
Is this what you were thinking? In this case, we need to have one server with Rest Server storing all the data. Hence duplicated data storage costs and also an issue when one server can’t store all the data since Ceph and co is not simple to set up and maintain.
Doing some more research. It looks like Backblaze and Wasabi have Object Locking and user permission to bypass this.
I’m not sure if this is a Backblaze and Wasabi thing or if it exists in S3 as well. It does sound like it is what we are looking for.
You have one server with S3 credentials and rclone configured to use these credentials. On this, you run rclone serve restic --append-only --addr <ip-addr>:<port> s3:repo
where <ip-addr> is the local ip adress and <port> is a free port.
From another host, you can then run restic -r rest:http://<ip-addr>:port backup ...
This will route the traffic over the rclone server and does not need any credentials (except the repo password) to run .
About the object locks: Yes, in principle this should work to secure your backups. However, the problem is that you cannot say a priori how long an object is needed for your backups. Restic only knows during forget/prune that a particular object is not longer needed and attempts to delete it immediately.
ahh, thank you. I didn’t know that one can run rclone as a proxy.
Restic with the bypassGovernence S3 user would run from another more isolated server. I assume the restic old backup deletion does not require any data outside of what is on S3, no?
Replacing an object in S3 with PutObject isn’t insecure, it’s how it works. This is why there’s object versioning, which is important for backups in case of malware encryption or similar. S3 includes a built in feature to remove old object versions, but this should be used after a restore test so you can be sure your backups work. I run manual restore tests every 3 months for my personal backups, using a script I run manually.
All restic commands only require the data stored within the repository (and the password of course). The locally stored cache is just a copy of some of the files in the repository and is exclusively used to optimize performance.
That is probably the only way to fully defend against an attacker who has full control over the system which is backed up (assuming that the backup is protected from deletion). After all an attacker could just backup garbage instead of the actual data.
If an attacker is just able to tamper with the backup storage but does not have the repository password, then tampering with a repository might be detectable by checking that no snapshots were deleted unexpectedly, that all object versions of a file have the same checksum and that check does not complain about problems. These three steps assume that the backup client created correct backups, if there’s a problem a that step then all bets are off.