we are some kind of master admins which are contracted by regular system admins for additional system administration.
I have to backup around 200 Hosts from 25 customers with numerous additional admins. Some hosts are very insecure and “admins” might do and will do weird things at any time.
So backups are very important for these machines.
I would love to backup all those machines with restic but i am afraid of repo security. What is a backup worth if an intruder can modify or delete the backup repo from a hacked host ? Do I have to add 200 users, one for every host we backup to separate the repos from each other ?
I can imagine backups via sftp or s3 or swift.
Is there any chance I can reliably run restic in such an environment ? Any ideas how to make this setup as secure as possible ?
As restic is doing push backups (as opposed to pull backups) the client/host will (almost) always have access to the repository. I think you have 2 viable options:
Use a policy to restrict access to the backend to only add files to all directories except lock. Pruning and thus deletion of files can be be handled by a privilged account.
Use regular snapthots (e.g. daily) on the backend to recover from “weird things” happening.
In any case you’ll want to have at least 25 repositories to separate data from your customers.
Thank you Alex for your hints and ideas. I will try to test a setup with read-only repo.
Also I have in mind some thing like a magical backup vm or backup container which somehow backups the client machine without giving the client machine any access to the backup repo. But I do not see how to access files on a running machine. I could export everything with NFS or SSH-Fuse but this is not what I like.
Do you see any way to reliably mount filesystems on running machines for backup purposes with restic ?
It’s always a good idea to prevent clients or attackers from deleting (or manipulating) data in the repository. But if you completly prevent read access to the repository your customers won’t be able to easily restore files themselves.
The issue Moritz linked to contains an example for pull backups with restic. This would be my prefered option in case you really want to go that route.
PS: I forgot to mention that rest-server also has an append-only mode which prevents clients/attackers from deleting any data.
Thanks for the Link, but my use case is different. I want a central and Secure “Backup Control Server” that has all Repo credentials and that Backups the 200 Client Machines to external Cloud Storage, not directly to the Backup Server.
So neither revers ssh nor read only rest-server do help in my case.
I think this approach is very interesting. Personally, I would not completely trust a third party with all my backups and keys to my backups (at the very end as client that’s my data); I get it, I really get that, for security reasons, you have to assume that you can’t trust the client but that’s as easy as revoke the key for the compromised client or clients. You have a read-only repository in a “cloud storage”, then you don’t really need to trust the client because it can’t delete anything. If anything goes wrong, revoke the client key. In the client you can create a new user just for restic with privileged access (assuming, of course, you’ll have access to the client), and just give access to that user to the restic binary and files containing the password and what you need to run restic from the client, so other users can’t access the binary or the data related to restic.
What I don’t really like about this approach (server making backups for clients) is that it can be misused. Maybe I’m wrong, but I don’t completely trust it.
Hmm,
maybe I have a lack of knowledge, but how realisitc is a real “read only repo in cloud storage” ?
Of course I have to prune files even in a read only repo on a daily basis to prevent the repo from growing and growing. Also with the repo credentials an intruder on the client can overwrite existing files as far as i know.
Would it be really realistic to set the perfect permissions on e.g. a swift storage to allow all good access and prevent all bad access ?
I think the @764287 first answer is the best. I don’t use Wasabi storage but I have tried it and as far as I know you can create different users with permissions (I use Wasabi as an example only but I think you can do that with other services too), so you can create a user who can just add data to the repo and use it for the client and another user with all permissions who will only be used in the server for all restic operations including forget, prune, etc. The “read-only” analogy I think is bad from my part because what you need is like an append only kind of thing. If you do regular snapshots I don’t think the data will be overwrite unless you keep only the latest snapshot which, for me, it is not the best thing to do because then you will only have one snapshot and if the latest snapshot is where the damaged data is, well, then the whole thing was worthless. As far as I know, restic will not overwrite data, it will add new data and if you keep all snapshots you will have the different versions of every file in every snapshot, so you can easily retrieve a damaged file in older snapshots. I don’t know if I’m wrong but a restic repository is read-only even when you mount it with fuse; you can’t really delete data unless you use restic forget --prune. You can’t also modify data, as far as I know. What I would do in your scenario is:
Create limited access users for the cloud storage.
Create a new user only for restic in the client encrypting the /home for that user so other users will not have access to the new “restic user” (a regular user will just see a bunch of encrypted files if they try to access to the new user’s home).
Put the restic binary in the new user’s home using these instructions, so the restic binary have access to everything.
Give the client access to the repo with the “limited user” only.
Use a more privileged user to manage repository from the server so it can actually apply forget policies and prune data.
Or, if you’re using a server to store the data you can use a REST-Server with the append-only capabilities, as mentioned in another comment.
I don’t know about swift storage. I also don’t know if I am totally right about my suggestion but I think that could work. Maybe later someone else can give you a better answer than this.
Can you maybe restate what your ultimate goals are and the reasoning behind it? I’m a bit confused why you say that none of the provided solutions fit your needs.
In my use case the clients that are backed up can be insecure.
In standard restic push configuration an intruder on the client can find the credentials of the cloud storage where the backups are, delete all backups there and then destroy or encrypt the client.
So in this standard configuration the backup seems useless in the worst case.
I want a backup where there is no way for an intruder to manipulate the backups on the cloud storage.
Also I want daily prunes of older backups not covered by the policy.
Last thing that would be nice is an easy management and monitoring of backups on 200 Hosts for 25 Customers.
If you only worry about intruders manipulating/deleting data then every solution posted in this thread should suffice. The most popular cloud solutions allow for creation of multiple users with different access rights.
Create a new bucket
Create a user which can read and add data to make backups
Create a maintenance user which has all rights and can thus forget and prune
If you are searching for a management console for restic you can take a look at Relica. I don’t know if it does everything you want but it might be your best bet.
We are currently and actively working on adding small-med business features, and we want to know more about their use cases, so feel free to shoot us an email: support@relicabackup.com with your requirements and we’ll see about getting it done!
Some backends (e.g. S3) have configurations that effectively disallow permanent deletions/overwrites by having a different user/role for creating backups. (In the case of S3, you only need to enable bucket versioning, and disallow the operations to delete specific versions. An attacker could then only insert a delete marker for objects, which is a trivial operation to reverse.)
Another option is to deploy the restic REST server with --append-only. Unfortunately, this requires e.g. an additional EC2 server. I’ve thought about writing an implementation of the REST server protocol using AWS API Gateway and Lambda, which would be a cost-effective way to implement append-only restic storage without requiring an always-on EC2.
Create limited access users for the cloud storage.
Create a new user only for restic in the client encrypting the /home for that user so other users will not have access to the new “restic user” (a regular user will just see a bunch of encrypted files if they try to access to the new user’s home).
Put the restic binary in the new user’s home using these instructions , so the restic binary have access to everything.
Give the client access to the repo with the “limited user” only.
Use a more privileged user to manage repository from the server so it can actually apply forget policies and prune data.
Some backends (e.g. S3) have configurations that effectively disallow permanent deletions/overwrites by having a different user/role for creating backups.
I was working to implement a similar idea about a year ago, but abandoned it when I realized that you can’t distinguish between creates and overwrites in S3. I also considered using Wasabi’s immutable buckets, but then we have a problem with repository locking.
While versioning does help, it creates some very annoying problems. I haven’t looked into the restic in-bucket format. Maybe it will, or can be easily modified to, play nice with a lifecycle policy to handle snapshot retention?
Other than using restic-server, I couldn’t find a maintainable solution.
Versioning is the way to do it. Just create a lifecycle policy that old versions get deleted after e.g. 7 days. Then you have 7 days to notice a repository compromise before the data is lost.
You could have a lifecycle policy for files under the snapshots directory, but note that this will only delete the snapshot record and not the actual directory/file contents. You’d still have to run a prune operation to garbage-collect the data directory.
Me too. Unfortunately, API Gateway is a bit overkill for this particular kind of solution, and deployment would not be straightforward.
If you run backups once a day, a compromise could be a t3.nano instance that only runs certain hours of the day, running the REST server in append-only mode and storing in an S3 bucket. You could launch the instance using a startup script that downloads and runs the REST server, and terminate the instance on shutdown. Then you don’t have to pay for the EBS 24/7.
One hour a day for a 30-day month at $0.0052/hr for t3.nano would be $0.16/mo.
Interesting that if you just delete the old snapshots, restic can clean up the unused blocks. There might be something useful there, though it doesn’t support the desirable daily, weekly, monthly setup.
I’m not familiar with restic-server’s ability to use S3 as a backend. Is this documented somewhere or would it be implemented through a connector?
Hmm, for me it still looks like all methods for making a restic repo secure from the client are complicated and awkward and complex.
So for this time we decided to not use restic for backups until there is a less complex way to prevent a restic repo being modified form the client.
It is a really a pity because restic is such a damned good piece of software
I have setup a backup server for my homelab (so security is more an academic exercise than a real need). What I have done is: I have create a set of scripts that receive as parameters two config files: one for the repo (one per host) and one for the target (NAS, BackBlaze, etc.).
In broad terms, what these scripts do is: they ssh into the host to be backed up and execute the restic command there. The configuration is extracted from the config files and used to build the single line command to be executed. The command string includes the restic command preceeded by setting some environment variables. For local repos, that command will use sftp to connect back to the backup server with a nologin user which can only access its own repo. This is probably not bullet proof. So, in theory, the host to be backed up is still able to destroy its own repo. But nothing else.
Also, if the backup server is compromised, it contains passwords for all the repos. Not a big deal for me but something to take into consideration.
BTW, I recently started using ansible and am considering migrating this solution to an ansible inventory and playbook which seem to be perfectly fit for this need. With the extra advantage of allowing me to use ansible vaults to store the sensible information currently vulnerable in the backup server.
PM me if you want me to share the scripts with you.