Restic 0.11.0

blog
1

Restic 0.11.0 was released today! For downloading the new release and to see a more detailed list of important changes, please head over to GitHub. If you already have restic (0.9.4 or later), you can use the self-update command to automatically download and verify the new release.

This is a companion discussion topic for the original entry at https://restic.net/blog/2020-11-05/restic-0.11.0-released/
7 Likes
2

Thanks for the work! Appreciate the early release with this feature. It would be great to support LVM snapshots for Linux.

3

Generic snapshot support for Linux is not going to be straightforward at all. There are multiple snapshot types each with their own caveats.

  • LVM snapshots
    • LVM operates at the block level. It is possible for one filesystem to be spread across multiple LVs (uncommon but useful in some cases) and for multiple filesystems to exist on the same LV (very uncommon). Detecting which LVs must be snapshotted for a given filesystem is not trivial.
    • There can be layers between the filesystem and the LV, such as crypto, md-raid, loopback, and other device-mapper mappings. It may not be possible to instantiate these layers on top of the snapshot automatically.
    • The VG needs free space to create a snapshot.
    • The size of the snapshot needs to be specified (this is the reservation for COW’d blocks).
    • The snapshots need to be mounted somewhere.
  • btrfs snapshots
    • There is no sane automatic way to determine where to put the snapshot. Within the subvolume being backed up would be the most straightforward approach but has its own potential issues.
  • ZFS snapshots
    • I lack the expertise to give details, though I assume it would mostly be similar to btrfs snapshots.
  • Probably some others.

This all gets way more complicated when multiple filesystems are involved:

  • For LVM, all of the issues are amplified: now we need to determine a set of distinct LVs that must be snapshotted and we need to create snapshots for each, specifying the size. It may not be possible to automatically determine sane sizes for each snapshot as this will depend on the expected write patterns to the LV during the backup. Each snapshot is itself atomic, but all snapshots together are not atomic with respect to each other.
  • For btrfs, subvolume snapshots are not recursive; any child subvolumes have to be snapshotted separately. Similar to LVM snapshots, each btrfs snapshot is itself atomic, but many snapshots cannot be made atomically as a group.

The simplest cases can probably be covered by adding restrictions:

  • No more than one filesystem is backed up (backup-with-snapshot would imply --one-file-system).
  • For LVM snapshots, the filesystem being backed up must exist directly on a LV; there cannot be any intermediate layers. A snapshot size or % of free space would need to be specified.

In all cases, paths would need to be adjusted to match the original path, and whatever was created would need to be torn down before restic terminates.

I would submit that it would be far better for restic to provide some example scripts that manage these processes (I could provide one for LVM and btrfs) and let system administrators adapt them to their own systems, for the following reasons:

  • Linux doesn’t have a VSS equivalent, meaning that applications don’t have a chance to make their on-disk data consistent before the snapshot is taken. Ideally this shouldn’t be necessary; if an application can survive an unexpected power down, it should be able to survive a snapshot. Nevertheless, without giving applications a chance to make their on-disk data consistent, the snapshot is “dirty.”
  • The system administrator really needs to understand what is going on under the hood, especially for the case where restic is unable to clean up the snapshot environment due to a hard crash or a power cut. There is no automatic LVM/btrfs snapshot cleanup on Linux; this has to be done manually. Blindly using such a feature leaves the sysadmin unaware of what must be done in this scenario. Failing to clean up snapshots can lead to the disk filling up (btrfs) or wasted reserved space (LVM) which could cause backups (or the whole system) to unexpectedly malfunction at a later date.
3 Likes
4

Hi Chris,
your analysis is very interesting and it’s the first serious analysis on what should be done to implement such a feature for a backup program I could found. Too often I see scarce interest on this topic in Linux backup software world :frowning:

In general I would say that restricting cases for LVM snapshots or BTRFS snapshots to the most simple one (i.e.: implying --one-file-system) would be already a good starting point to cover simple backup scenarios (probably the most frequent ones?).

Regarding this:

There can be layers between the filesystem and the LV, such as crypto, md-raid, loopback, and other device-mapper mappings. It may not be possible to instantiate these layers on top of the snapshot automatically.

Could you elaborate more on this with an example?

5

For example, if you have ext4 on top of LUKS, which is on top of LVM, then restic needs to understand the device-mapper layers to see that LUKS is between ext4 and LVM. If it determines this, it can take an LVM snapshot of the LUKS container, but then it can’t really automatically open the LUKS container without a passphrase.

6

I see now, thank you. I admit I never tried and I naively didn’t think about the fact that in such a situation the snapshotted LUKS container would need to be opened.
The good news is that perhaps the most widespread configuration is LVM on top of LUKS rather than the contrary, at least this is what you get when you perform an encrypted+LVM installation in Debian/Ubuntu.

7

Right, usually there isn’t anything between LVM and the filesystem.

I have one case where I have LUKS on top of LVM, but this is because I give LVs to a VM as its virtual disks, and the guest does LUKS on top – so neither system is fully aware of all of the layers. The host only knows about LVM and LUKS (but the LUKS container is not open), and the guest only knows about LUKS and the filesystem. Of course, restic wouldn’t be able to manage any snapshotting here because it can’t see the whole picture, so it’s not a great example.

Anyway, I think if restic is going to do any kind of automatic snapshots, it would make the most sense to at least start with filesystems that natively support snapshots (btrfs and ZFS) as they are the most straightforward. Restic doesn’t have to know or care about any storage layers whatsoever to make this work.

Supporting LVM snapshots will require probing the block storage layout. This is more complicated, harder to get right, and in the end restic could probably only support the simplest of layouts anyway.

1 Like
8

Maybee veeamsnap should be supported. It is gpl-2 license:

“This kernel module implements snapshot and changed block tracking functionality used by Veeam Agent for Linux – simple and FREE backup agent designed to ensure the Availability of your Linux server instances, whether they reside in the public cloud or on premises.”

9

Unless this feature is integrated into mainline Linux (so many people can use it without having to build their own kernel module) I don’t think we have the resources to support it. Sorry about that. :slight_smile:

1 Like