Founder of BorgBase.com here. We’ve been offering hosting of Borg repositories since 2018 and just added Restic as additional format/tool. Not because we don’t like Borg any more, but to give our users some choice.
We also like to engage with the community and build lots of open source stuff around the tools we offer. For Borg we maintain Vorta, a popular GUI that’s also shipped with Debian and Ubuntu and an Ansible role.
So I’m excited to make our Restic repository feature generally available. As with every new feature, there will be things to improve, so I’m also looking to get some feedback on what’s missing right now. Currently we use Nginx + HTTP/2 + Restic REST. I already did a quick benchmark with B2 and we’re about 3x faster:
Times are for running each command and last the data transfer. Test data: total 170864 files 41.103 GiB, /var folder with logs, system files and Prometheus and SQLite databases. My goal was to benchmark raw upload speed, deduplication and pruning. Commands in detail:
B2 maxes out around 250Mbit, while we reach up to 850 Mbit. Hence the upload takes 3x longer.
While pruning around 85 MB are downloaded for “repack”. Using the Restic REST server, we don’t see this with BorgBase. (we also don’t charge for bandwidth, so it wouldn’t matter much)
So my questions:
Is there anything that could be tuned even more to get better performance for Restic? Architecture-wise and so on?
What else would you like to see for hosted Restic repositories? Currently we have monitoring (for stale repos) and high-level logs that tells you about backup runs and data added.
Would you use our service over generic providers? (better performance, lower per-GB cost, no API or bandwidth fees, built-in monitoring and logs)
What’s missing in the Restic eco system that could be built or sponsored right now?
That’s all for now. Hope this is useful. Happy to answer any questions. For more, head over to BorgBase.com. First 10 GB are free forever and I can give free larger plans to open source contributors of the project (PM with Github username).
While I do appreciate the questions, most of the above post reads like an advertisement, in particular since it’s about a paid product rather than just an open source and free community thing.
Sorry if it gets across like that. Just excited to get this out of the door and see such good performance. Also couldn’t find a better category, but do feel free to change it.
I’m hoping to get some ideas for future open source projects. Like we already do with Borg. Or opportunities for sponsorships.
The official docs also contain a range of paid storage options. So I assumed it’s OK to disucss this. Backups need to go somewhere
Funny story actually: Someone already forked Vorta, the GUI we maintain for Borg and tried to adjust it for Restic. Sadly they abandoned it after a short while.
First I have to say that the main difference between restic and borg IMO is that the restic repo format does allow to use generic storage providers while the borg repo format does impose extra difficulties when using with storage providers or with multiple clients. This is why borg in many use cases needs the server component to provide the borg API whereas restic doesn’t.
For you, this is a bad message as you are competing with much more other players…
Besides optimizations on the client side which I think are not your top prios, it always boils down to provide fast file access on repository files. The REST API is already pretty good. I wrote an alternative in Rust - https://github.com/aawsome/rust-server - which might perform better CPU-wise than the go rest-server when you have many connections. But I’d think the focus here usually is I/O. So I would advise you to use the better tested rest-server.
Honestly, I myself don’t care about extra monitoring or logs - I can do all this on the client.
About what’s missing I must say that I made rustic exactly to add things I am missing in restic. These are mainly:
support for cold storage
lock-free operations
support for a config file
But as said before, these are all things that do not impose special needs for a storage provider. For you it might be interesting to also offer cold storage. In my view this is THE killer feature for large data as it allows to use really cheap storage backends - if you can tolerate warm-up times for your restore.
Thanks for weighing in and kudos on your work with Rest and Rustic. Pretty cool to see so many competing implementations and explore new paths that way. One of the Borg GUIs we sponsor is also using Rust (Pika Backup for Gnome that is).
Regarding the number of storage providers for Borg vs Restic, it’s probably similar. There were already many SSH/SFTP-based providers when we started and it’s very easy for them to add one more binary. So they tended to offer all of them. Having some competition is good really, like between Go- and Rust implementations of a similar application. The market is large enough and we have differentiated the service by helping people with their setup and strategy, as well as supporting or building many useful FOSS tools that make the the core application more useful.
I’ll definitely try to contribute to the client development, after brushing up my Go skills. Didn’t use them for a while. For the server side, we keep improving our setup with every server generation. The latest one uses a large NVMe cache and cheaper spinning rust behind that. Hoping to use this setup more widely, also for Restic repos.
Our main focus is operative backups, rather than archiving. I believe those cases have different requirements and picking one keeps the focus. Will still read up on how Rustic supports cold storage to get a better idea.
Which restic version did you use for testing? 0.14? That version contains quite a few performance optimizations like improved parallelism and larger pack file sizes which help with saturating fast network connections. You could try, whether increasing the number of connections using the -o rest.connections=10 (default is 5) or --pack-size 128 (default is 16MB) helps. Both options only make sense if a lot of upload bandwidth is available.
There are already a few more performance optimizations e.g. for small files included in the development version of restic (aka. the master branch).
Yes, I used 0.14 for testing. I meant optimizations on the server-side for Restic REST server or Nginx. So our users get better performance.
Still good to know available client-side optimizations. Could even go into the docs. For Borg, I collected those in our own docs for a while and then we added them back to the official Borg docs, since they were quite generic.
Append only is coming in the next days. It’s already there in the backend. Just seeing on how to best add it together with SFTP (for exporting raw repos).
Looks like having keys is a benefit for some and a disadvantage for others.
SSH for Restic would mean SFTP, right? We already have that for importing and exporting raw repos. But don’t promote other uses because it can’t be made append-only easily. As opposed to Restic REST server. SFTP is also slower according to the docs. So you probably want HTTP in most cases.
Since you can easily add the username and password in the URL, it’s also quite easy to use. For added security, I want to add an option to refresh those logins.
Yes, not having to rely on password auth would be nice, but I don’t necessarily see that as a reason for needing SSH…
Since it sounds like there is an Nginx placed in front of the Rest Server I presume that it would be possible to implement TLS client auth?
What would be especially nice then would be to be able to upload a CA cert, and then simply provide per repository subject conditions. That way one would be able to automatically rotate TLS keys, re-provision instances, etc without having provide BorgBase with a new client cert each time.
Very interesting idea. Already looking into it and totally doable. You’re right, only the CA cert is needed on the backup server, so you could rotate client certs yourself.
I did play with the setup several years ago, but for some reason it didn’t catch on. I remember only this one CA used them for login, StartSSL or something.
Append-only setting is now available. Looks like this (predictably) :
$ restic forget a1025157
repository 587dd50d opened (repository version 2) successfully, password is correct
Remove(<snapshot/a1025157a6>) returned error, retrying after 552.330144ms: blob not removed, server response: 403 Forbidden (403)
Remove(<snapshot/a1025157a6>) returned error, retrying after 1.080381816s: blob not removed, server response: 403 Forbidden (403)
Also added some new icons to quickly see which backup format is used and if append-only is enabled.
That looks nice, thank you for adding restic support. It would be great if the restic repos behaved more like borg repos. It would be great to be able to create “keys” (basically username/password pairs) and assign them to restic repos and have append-only not as a repo setting but rather dependent on the key.
Already looking into it and totally doable. You’re right, only the CA cert is needed on the backup server, so you could rotate client certs yourself.
: