Restic repositories on BorgBase.com + Benchmark

Hey all,

Founder of BorgBase.com here. We’ve been offering hosting of Borg repositories since 2018 and just added Restic as additional format/tool. Not because we don’t like Borg any more, but to give our users some choice.

We also like to engage with the community and build lots of open source stuff around the tools we offer. For Borg we maintain Vorta, a popular GUI that’s also shipped with Debian and Ubuntu and an Ansible role.

So I’m excited to make our Restic repository feature generally available. As with every new feature, there will be things to improve, so I’m also looking to get some feedback on what’s missing right now. Currently we use Nginx + HTTP/2 + Restic REST. I already did a quick benchmark with B2 and we’re about 3x faster:

To BorgBase

init: 0m4.085s
backup /var: 3m26.347s
backup /var/log: 0m4.790s
forget: 0m46.537s
RX/TX: 124.12MiB    11.23GiB

To B2

init: 0m5.447s
backup /var: 11m3.706s
backup /var/log: 0m13.111s
forget: 1m2.112s
RX/TX: 236.77MiB    11.03GiB

Times are for running each command and last the data transfer. Test data: total 170864 files 41.103 GiB, /var folder with logs, system files and Prometheus and SQLite databases. My goal was to benchmark raw upload speed, deduplication and pruning. Commands in detail:

restic init
restic backup /var
restic backup /var/log
restic forget --keep-last 1 --prune

Main things, I see here:

  • B2 maxes out around 250Mbit, while we reach up to 850 Mbit. Hence the upload takes 3x longer.
  • While pruning around 85 MB are downloaded for “repack”. Using the Restic REST server, we don’t see this with BorgBase. (we also don’t charge for bandwidth, so it wouldn’t matter much)

So my questions:

  • Is there anything that could be tuned even more to get better performance for Restic? Architecture-wise and so on?
  • What else would you like to see for hosted Restic repositories? Currently we have monitoring (for stale repos) and high-level logs that tells you about backup runs and data added.
  • Would you use our service over generic providers? (better performance, lower per-GB cost, no API or bandwidth fees, built-in monitoring and logs)
  • What’s missing in the Restic eco system that could be built or sponsored right now?

That’s all for now. Hope this is useful. Happy to answer any questions. For more, head over to BorgBase.com. First 10 GB are free forever and I can give free larger plans to open source contributors of the project (PM with Github username).

Thanks!
Manu

3 Likes

While I do appreciate the questions, most of the above post reads like an advertisement, in particular since it’s about a paid product rather than just an open source and free community thing.

1 Like

Sorry if it gets across like that. Just excited to get this out of the door and see such good performance. Also couldn’t find a better category, but do feel free to change it.

I’m hoping to get some ideas for future open source projects. Like we already do with Borg. Or opportunities for sponsorships.

The official docs also contain a range of paid storage options. So I assumed it’s OK to disucss this. Backups need to go somewhere :grimacing:

1 Like

Funny story actually: Someone already forked Vorta, the GUI we maintain for Borg and tried to adjust it for Restic. Sadly they abandoned it after a short while.

First I have to say that the main difference between restic and borg IMO is that the restic repo format does allow to use generic storage providers while the borg repo format does impose extra difficulties when using with storage providers or with multiple clients. This is why borg in many use cases needs the server component to provide the borg API whereas restic doesn’t.
For you, this is a bad message as you are competing with much more other players…

Besides optimizations on the client side which I think are not your top prios, it always boils down to provide fast file access on repository files. The REST API is already pretty good. I wrote an alternative in Rust - https://github.com/aawsome/rust-server - which might perform better CPU-wise than the go rest-server when you have many connections. But I’d think the focus here usually is I/O. So I would advise you to use the better tested rest-server.

Honestly, I myself don’t care about extra monitoring or logs - I can do all this on the client.

About what’s missing I must say that I made rustic exactly to add things I am missing in restic. These are mainly:

  • support for cold storage
  • lock-free operations
  • support for a config file

But as said before, these are all things that do not impose special needs for a storage provider. For you it might be interesting to also offer cold storage. In my view this is THE killer feature for large data as it allows to use really cheap storage backends - if you can tolerate warm-up times for your restore.

1 Like

Hey Alex,

Thanks for weighing in and kudos on your work with Rest and Rustic. Pretty cool to see so many competing implementations and explore new paths that way. One of the Borg GUIs we sponsor is also using Rust (Pika Backup for Gnome that is).

Regarding the number of storage providers for Borg vs Restic, it’s probably similar. There were already many SSH/SFTP-based providers when we started and it’s very easy for them to add one more binary. So they tended to offer all of them. Having some competition is good really, like between Go- and Rust implementations of a similar application. The market is large enough and we have differentiated the service by helping people with their setup and strategy, as well as supporting or building many useful FOSS tools that make the the core application more useful.

I’ll definitely try to contribute to the client development, after brushing up my Go skills. Didn’t use them for a while. For the server side, we keep improving our setup with every server generation. The latest one uses a large NVMe cache and cheaper spinning rust behind that. Hoping to use this setup more widely, also for Restic repos.

Our main focus is operative backups, rather than archiving. I believe those cases have different requirements and picking one keeps the focus. Will still read up on how Rustic supports cold storage to get a better idea.

A lot of the discussions on cold-storage support can also be found here Discussion: How to support cold storages? · Issue #3202 · restic/restic · GitHub.

Which restic version did you use for testing? 0.14? That version contains quite a few performance optimizations like improved parallelism and larger pack file sizes which help with saturating fast network connections. You could try, whether increasing the number of connections using the -o rest.connections=10 (default is 5) or --pack-size 128 (default is 16MB) helps. Both options only make sense if a lot of upload bandwidth is available.

There are already a few more performance optimizations e.g. for small files included in the development version of restic (aka. the master branch).

Yes, I used 0.14 for testing. I meant optimizations on the server-side for Restic REST server or Nginx. So our users get better performance.

Still good to know available client-side optimizations. Could even go into the docs. For Borg, I collected those in our own docs for a while and then we added them back to the official Borg docs, since they were quite generic.

Hi,
Append-only backups would be nice.
With the setup now a hacker can delete the backups.
Love the web gui and ease to setup!

Hey Jef,

Thanks for testing it out!

Append only is coming in the next days. It’s already there in the backend. Just seeing on how to best add it together with SFTP (for exporting raw repos).

Do you plan ssh as backend as well?

Currently not. Wouldn’t HTTP/2 be superior to SSH? We have SSH for Borg repos and especially new users often have issues managing their keys.

Or do you see any particular benefit in having Restic+SSH?

We do have SFTP for importing and exporting raw repos.

I was just curious. One benefit, I could think of would be key management of SSH instead of a simple HTTP Auth password.

Looks like having keys is a benefit for some and a disadvantage for others. :laughing:

SSH for Restic would mean SFTP, right? We already have that for importing and exporting raw repos. But don’t promote other uses because it can’t be made append-only easily. As opposed to Restic REST server. SFTP is also slower according to the docs. So you probably want HTTP in most cases.

Since you can easily add the username and password in the URL, it’s also quite easy to use. For added security, I want to add an option to refresh those logins.

Yes, not having to rely on password auth would be nice, but I don’t necessarily see that as a reason for needing SSH…

Since it sounds like there is an Nginx placed in front of the Rest Server I presume that it would be possible to implement TLS client auth?

What would be especially nice then would be to be able to upload a CA cert, and then simply provide per repository subject conditions. That way one would be able to automatically rotate TLS keys, re-provision instances, etc without having provide BorgBase with a new client cert each time.

Very interesting idea. :thinking: Already looking into it and totally doable. You’re right, only the CA cert is needed on the backup server, so you could rotate client certs yourself.

I did play with the setup several years ago, but for some reason it didn’t catch on. I remember only this one CA used them for login, StartSSL or something.

Feel free to give me a ping in case you’d like some specific feedback, help testing or whatnot!

Append-only setting is now available. Looks like this (predictably) :smile::

$ restic forget a1025157
repository 587dd50d opened (repository version 2) successfully, password is correct
Remove(<snapshot/a1025157a6>) returned error, retrying after 552.330144ms: blob not removed, server response: 403 Forbidden (403)
Remove(<snapshot/a1025157a6>) returned error, retrying after 1.080381816s: blob not removed, server response: 403 Forbidden (403)

Also added some new icons to quickly see which backup format is used and if append-only is enabled.

1 Like