restic seems to have taken over the open source personal backup world, and I’ve been using it happily myself for a few years. Thanks to the developers for your hard work. However, I’ve run into a couple problems. Here’s my current setup:
3TB (~50% full) FreeNAS VM with mirrored ZFS, accessed over NFS
restic to back that up to a USB HDD.
rclone to copy the USB restic backup to Backblaze B2
Most of my data is images, videos, and music files that don’t change much after being added (though they might be moved around occasionally).
Here are the problems I’ve encountered:
It’s very slow. Takes about 4 hours to run restic, and that doesn’t include the time to upload via rclone
I’m concerned that a working version of restic is required to restore my data, rather than simply being stored as flat files.
In my mind, my ideal backup tool would work like this:
Minimal dependencies. Preferably just stores as timestamped directories with hard links for files that haven’t changed, and updated versions for files that have. I don’t mind using extra storage from not compressing, using file-level deduplication, etc. As long as it’s fairly efficient (ie doesn’t copy the entire filesystem every snapshot).
Fast. I don’t need encryption, and if I change my mind about that I’d prefer to use disk-level encryption rather than having it built into my backup tool.
Simple. I don’t need features such as fuse mounting.
Are there any tools out there that work like this, or is there a way to use restic more in this way?
I’ve been reading up on restic vs borg. So far borg seems even more complicated. It relies on a server component, correct? I like that restic stores everything in a repo directory that you can then rclone around as necessary. But I should definitely at least give borg a try and see if it addresses some of my concerns.
Yeah. In fact borg needs to be installed on the destination. You don’t need to have borg running all the time. For example, when you initiate a backup over SSH it will automatically start borg in ‘serve’ mode. What’s nice with borg installed on the destination is that you can check the repository by running borg on the destination.
One drawback with borg is that it does not support a lot of backend and to my knowledge there is no rclone support.
Check out Dirvish, a wrapper for rsync. It’s fairly simple and creates directories with hard links. You don’t need tools to restore. No deduplication as far as I know though.
I would expect that to be due to the USB HDD. Not sure what you expect Do you know if you have AES-NI in your CPU? If so, there being encryption shouldn’t have any relevant performance penalty on the processing.
Right, this is something I’ve considered too. But the restic repository format is highly documented, and there’s really no reason why you wouldn’t be able to run restic when you need to, so I don’t think this will ever be a problem in practice. Also you have to remember that using restic gets you so much more that even if you’d rather have plain files, it might be worth it.
As you probably already know, restic only uploads to the repository the parts of your files that it hasn’t already uploaded before, de-duplication. So it’s efficient alright. And it sure as heck has minimal dependencies, restic is just the restic binary.
If you truly want a file-level only backup tool, then what @nicnab mentioned and similar software is what you’d have to use. Both they and restic has their pros and cons. You will have less opportunity to verify the integrity of your backups, that’s one con.
What’s the point of that? It’s more complicated to have disk-level encryption, especially when you consider using an offsite repository, and in that latter case it can be less secure due to having to manage mounting the disk and entering the key. Anyway, as long as you have AES-NI in your CPU you shouldn’t notice any performance penalty for there being encryption going on.
No need to use it
In general, I love that restic is just one single binary, no dependencies, that it doesn’t require configuring stuff, I can just ad-hoc back things up with it, I can mount the repository if I need to browse my backup, it’s fast and I feel it’s reliable and secure. Combined with rest-server it’s even better! Not sure what else I’d want really.
If you want faster backups, zfs snap replication is probably fastest.
If you want a tool that works to your description then probably rsnapshot (yuk).
I haven’t done benchmarks but I “feel” like borg is faster than restic.
If you have a lot of files, rsnapshot is going to be very slow. I personally won’t touch anything rsync for backups anymore. The technology is way past that now IMO.
Ooooh don’t say that. I have had a long lasting love relationship with rsync and we share so many good memories. I have used dirvish for a while which is an rsync script using hard links and rsnapshot seems to be similar but even simpler - and still maintained.
That ZFS thing looks promising but personally, I’m a huge fan of simplicity and ZFS - even by itself - just ain’t simple. It just makes me uneasy if it’s really hard to understand even the basic concepts behind something. restic is pretty easy to understand even for me and happens to just work. borg I haven’t tried.
If you’re going for simple and unencrypted, I would recommend good old rsync --link-dest or rsnapshot (based on hardlinks). I used custom Bash scripts based on rsync before, but one downside is that rsync doesn’t detect renames, so if you shuffle your files around a bit, you end up storing the same stuff multiple times.
As for fast, I consider restic to be quite fast and efficient, so I wonder where the bottleneck is in your setup? What is the raw speed of writing to your USB disk?
If you want to have a flat-file backup with snapshots, have a loot at “Back in Time”. It uses hard links to deduplicate files. Restore is trivial, you just have snapshot directories with the full date as names and then your whole directory tree below that.
You restic… Then rclone? what’s wrong with restic for all purposes? From: “Supports storage on various cloud services natively and through rclone.”
rsync, has long been, and will continue to be the baseline for file transfer… It’ll survive anything, file level, incremental, preinstalled; it just is
Neither is rclone if you want to be precise about the matter, yet restic relies upon it, which comes from? Rsync. The OPs concern was longevity of maintenance. If you want to be sure you can access your files in an arbitrary number of years, rsync is the most likely tool to enable that. Why are rclone and rsnapshot and numerous other backup tools and utilities based upon or part thereof, rsync? Because it has stood the test or time and is highly flexible and well maintained. It on its own is not a utility, it is a tool. But I never claimed it was anything, so I’m not sure what your point is, aside from disagreeing with me with passive aggression. Latest craze sweeping the internet apparently
Restic does not rely on rclone, you can use restic without rclone. You use restic with rclone when you want to store your repository on one of the services that rclone supports but that is not in the list of supported backends out of the box. However, most common storage services out there these days support S3, so the S3 backend applies to a lot of them. But there are indeed some use cases where rclone is needed as well
Yes, plain copies of the files will of course always be the most straight forward way of accessing the files at a later date. Storing them in restic or other backup software formats adds an additional layer. In restic’s case this format is very clearly documented, and the idea behind that is that backups will not be locked into some closed format that is hard to work with - instead at any time in the future, even if restic were to not be around, it will be clear how the information is stored, and access to it can be implemented. Then again it’s very unlikely that one cannot get a hold of a copy of restic for the foreseeable future
I sincerely have no idea why you think I was being passive aggressive. If you read what I wrote I only pointed out that rsync is not a backup software (as it’s a file synchronization software). I did this so that any less knowledgable readers won’t accidentally read this thread and think that they can substitute a backup software like restic with rsync and expect the same features. They are similar, but there are differences that are relevant enough to avoid mixing them up, I think. Either way, I was not looking to disagree with you.
I did not attack you in any way and I would be happy if you don’t suggest I’m on some spree to spread aggression on the Internet, when I was just making a remark. Let’s be friends!
Actually apart from starting with the letter ‘r’, rclone and rsync are completely unrelated. Rsync is written in C by Andrew Tridgwell. Rclone is written in go by Nick Craig-Wood.
Another point of difference with rsync is that it needs to execute on the sending and receiving hosts. This is not possible with object store cloud services. Rclone implements each of the cloud services client apis itself. Rsync requires a filesystem.
Both tools rock though.
If you’re an oldie like me, you might remember many of the BSD ‘r’ commands of the day before OpenSSH, like rcp rexec rsh and rdist which was the go to tool before rsync for synchronising files between 2 or more hosts. All secured through your .rhosts config file in your home directory. No encryption or passwords.
Do you know if you have AES-NI in your CPU? If so, there being encryption shouldn’t have any relevant performance penalty on the processing.
