restic seems to have taken over the open source personal backup world, and I’ve been using it happily myself for a few years. Thanks to the developers for your hard work. However, I’ve run into a couple problems. Here’s my current setup:
3TB (~50% full) FreeNAS VM with mirrored ZFS, accessed over NFS
restic to back that up to a USB HDD.
rclone to copy the USB restic backup to Backblaze B2
Most of my data is images, videos, and music files that don’t change much after being added (though they might be moved around occasionally).
Here are the problems I’ve encountered:
It’s very slow. Takes about 4 hours to run restic, and that doesn’t include the time to upload via rclone
I’m concerned that a working version of restic is required to restore my data, rather than simply being stored as flat files.
In my mind, my ideal backup tool would work like this:
Minimal dependencies. Preferably just stores as timestamped directories with hard links for files that haven’t changed, and updated versions for files that have. I don’t mind using extra storage from not compressing, using file-level deduplication, etc. As long as it’s fairly efficient (ie doesn’t copy the entire filesystem every snapshot).
Fast. I don’t need encryption, and if I change my mind about that I’d prefer to use disk-level encryption rather than having it built into my backup tool.
Simple. I don’t need features such as fuse mounting.
Are there any tools out there that work like this, or is there a way to use restic more in this way?
I’ve been reading up on restic vs borg. So far borg seems even more complicated. It relies on a server component, correct? I like that restic stores everything in a repo directory that you can then rclone around as necessary. But I should definitely at least give borg a try and see if it addresses some of my concerns.
Yeah. In fact borg needs to be installed on the destination. You don’t need to have borg running all the time. For example, when you initiate a backup over SSH it will automatically start borg in ‘serve’ mode. What’s nice with borg installed on the destination is that you can check the repository by running borg on the destination.
One drawback with borg is that it does not support a lot of backend and to my knowledge there is no rclone support.
Check out Dirvish, a wrapper for rsync. It’s fairly simple and creates directories with hard links. You don’t need tools to restore. No deduplication as far as I know though.
I would expect that to be due to the USB HDD. Not sure what you expect Do you know if you have AES-NI in your CPU? If so, there being encryption shouldn’t have any relevant performance penalty on the processing.
Right, this is something I’ve considered too. But the restic repository format is highly documented, and there’s really no reason why you wouldn’t be able to run restic when you need to, so I don’t think this will ever be a problem in practice. Also you have to remember that using restic gets you so much more that even if you’d rather have plain files, it might be worth it.
As you probably already know, restic only uploads to the repository the parts of your files that it hasn’t already uploaded before, de-duplication. So it’s efficient alright. And it sure as heck has minimal dependencies, restic is just the restic binary.
If you truly want a file-level only backup tool, then what @nicnab mentioned and similar software is what you’d have to use. Both they and restic has their pros and cons. You will have less opportunity to verify the integrity of your backups, that’s one con.
What’s the point of that? It’s more complicated to have disk-level encryption, especially when you consider using an offsite repository, and in that latter case it can be less secure due to having to manage mounting the disk and entering the key. Anyway, as long as you have AES-NI in your CPU you shouldn’t notice any performance penalty for there being encryption going on.
No need to use it
In general, I love that restic is just one single binary, no dependencies, that it doesn’t require configuring stuff, I can just ad-hoc back things up with it, I can mount the repository if I need to browse my backup, it’s fast and I feel it’s reliable and secure. Combined with rest-server it’s even better! Not sure what else I’d want really.
If you want faster backups, zfs snap replication is probably fastest.
If you want a tool that works to your description then probably rsnapshot (yuk).
I haven’t done benchmarks but I “feel” like borg is faster than restic.
If you have a lot of files, rsnapshot is going to be very slow. I personally won’t touch anything rsync for backups anymore. The technology is way past that now IMO.
Ooooh don’t say that. I have had a long lasting love relationship with rsync and we share so many good memories. I have used dirvish for a while which is an rsync script using hard links and rsnapshot seems to be similar but even simpler - and still maintained.
That ZFS thing looks promising but personally, I’m a huge fan of simplicity and ZFS - even by itself - just ain’t simple. It just makes me uneasy if it’s really hard to understand even the basic concepts behind something. restic is pretty easy to understand even for me and happens to just work. borg I haven’t tried.
If you’re going for simple and unencrypted, I would recommend good old rsync --link-dest or rsnapshot (based on hardlinks). I used custom Bash scripts based on rsync before, but one downside is that rsync doesn’t detect renames, so if you shuffle your files around a bit, you end up storing the same stuff multiple times.
As for fast, I consider restic to be quite fast and efficient, so I wonder where the bottleneck is in your setup? What is the raw speed of writing to your USB disk?
If you want to have a flat-file backup with snapshots, have a loot at “Back in Time”. It uses hard links to deduplicate files. Restore is trivial, you just have snapshot directories with the full date as names and then your whole directory tree below that.
Do you know if you have AES-NI in your CPU? If so, there being encryption shouldn’t have any relevant performance penalty on the processing.