Restic vs ZFS differences

Hello everyone,

I’m a new user to restic as well as ZFS. I don’t know if this is a comparison similar to apples and oranges but I would like to know what differences are between restic and ZFS. I understand that ZFS is a file system and restic is a backup solution. But I noted that both restic and ZFS can actually provide similar functionality too! E.g, both allow taking snapshots with minimum space overhead, both allow browsing through snapshots to observe the content of snapshotted files.

In my use case, I would like to have a snapshotted multi- terabyte ZFS file storage backed up to Wasabi S3 using restic. This way I will have multiple snapshots. Some snapshots will be in the ZFS and a lot more in the restic S3 repo.

What are some excellent cases both restic and ZFS are used for personal file storage? For me, snapshots are very important, mounting them, browsing through them is very important. S3 provider not being able to read my S3 storage due to encryption is very important.

What is your setup? How do you effectively use ZFS + Restic?

Would appreciate some light shred on this idea…

Thanks to everyone who make restic possible.

1 Like

You seem to already understand the technical aspects of it, so there’s not much “help” to give here. If you’re just wanting to know how people use restic and ZFS together, that’s more of a discussion.

Personally I use ZFS to store my backups. That is, I back up to a server that uses ZFS as the filesystem for the storage where it keeps the repositories.

2 Likes

But I am wondering if there is a way I could combine restic and ZFS in an effective manner.

Your question is rather unclear. You have to be more specific.

If you are asking whether you can use ZFS snapshots in restic or vice-verse, then no, not in any sense that gets you some benefit in terms of performance or storage or similar. They are two different things.

I’m not sure what answer you are looking for. If you still think it’s unclear, please be more specific.

My backup script on our production servers makes use of LVM snapshots to ensure that backups are effectively atomic, point-in-time snapshots of the system as a whole. ZFS snapshots could almost certainly provide similar atomicity guarantees for backups.

The script I wrote makes a snapshot of the volume to be backed up and sets up a chroot environment. Within that environment, a second instance of the database server is launched and a MySQL database dump is taken from that. (We are still using MyISAM tables in a lot of cases, which require a database-wide exclusive lock for the duration of the dump to obtain a consistent dump. This approach lets us take a database dump without locking the production server for 15+ minutes.) Finally, a backup of the entire filesystem (sans the MySQL datadir) are taken as well. When this is all done, the chroot environment is unmounted and the LVM snapshot is deleted.

I suspect that most people naively run restic against / which may work in many cases. However, it is not atomic, and can result in snapshots that are not internally consistent. The longer it takes the backup to run, the worse the problem is.

I’d strongly advise you to make use of ZFS snapshots for the same purpose that I use LVM snapshots. Otherwise, you have no guarantees that your backup is even usable unless you are running a pure file server.

Exact, restic is fine for static file backups. Database backup is a very tricky stuff since you have to make sure everything is consistant or you wont be able to use your backups.

I have experience with both PG & MySQL.

For PG I use barman for years without any problems yet (+25 databade of several hundreds of GB). This act like timemachine since you can restore the database from any time (as long as you have the WAL files).

For MySQL I have my production ready / heavily tested script: https://github.com/renard/mysql-backup. I use is against several hundreds of mysql databases some of them with several hundreds of GB. It works with both ZFS (recommended) and LVM to make sure all backups are consistant. Like @cdhowie does, it checks the data consistency before performing the archive. It can be used to do binary backups (recommended for fast restoration) or text dump (discouraged on large databases). Unlike barman, you can only restore database at time of backup or you have to manually play with binlogs.

Honestly, taking consistent database backups is the easy part - you just dump it. Making sure your backups files and database and other data sources are in sync when you take the backup, that’s trickier.

Just a note since I do this. I have a freenas box that takes hourly snapshots. These snapshots are pruned such that I keep 1 snapshot per hour in the last day, 1 snapshot per day in the last 2 weeks, and 1 snapshot per week for the last year.

Separately I do restic backups. I run the restic backups only daily instead of hourly. Once per day, I take a zfs snapshot named @restic and clone the snapshot and mount it. Then I use restic to backup to the cloud. Finally, I clean up the temporary restic snapshots.

This is the core commands run by my script, though I do some extra stuff for cleanup and to only backup a subset of the data. I also prune once a week:
My zpool name is data1

zfs snapshot -r data1@restic
for SNAP in $(zfs list -o name -t snapshot | grep @restic)
do
CLONE=“data1/clone${SNAP:5:-7}”
zfs clone ${SNAP} ${CLONE}
done

restic -r repo backup --exclude-caches /mnt/data1/clone

zfs destroy -Rr data1@restic
Blockquote

Yep, that’s part of the reason why I use LVM snapshots. Of course, this requires that all files (database and other files) live on the same volume; while LVM snapshots are atomic, multiple LVM snapshots of different volumes are not atomic with respect to each other.

It also requires that your application is aware of backings being made so the combination of database and files can be atomic.

Well, that’s not actually possible unless the underlying filesystem supports transactions. More likely, the application would need to order filesystem operations in such a way that it can detect and recover from a backup (which would look identical to power loss from the application’s perspective). Essentially, if an application can recover from a power loss, it can also “recover” from a filesystem-level atomic snapshot. (I think this is what you meant.)

This has derailed. I was merely trying to point out that dumping your database properly and taking a snapshot with your filesystem (be it on LVM or be it ZFS or something else) and backing all that up will not necessarily give you a consistent backup, in cases where you have applications that work with data in both a database and the filesystem. In order to back such things up you pretty much need the application to be backup-aware/have a feature in which it allows consistent/atomic backups to be taken.