Best Way To Do a Full System Backup

[Grass]

Introduction

Hey, I’ve used Restic to backup some stuff before and now I’m thinking of using it to backup my whole filesystem as part of my Linux configuration.

Solutions I Found

I’ve read the docs, but it doesn’t address the issue of ongoing filesystem writes during a backup, which I heard can cause issues with the backup? To that end, I saw someone else’ setup where we do

export RESTIC_FEATURES="device-id-for-hardlinks=true" restic backup ...

on a fresh BTRFS snapshot.

Queries

• Is this the best way to do a full filesystem backup?

• A cursory reading of issue 3041 and PR 4006 seems to suggest enabling the feature flag RESTIC_FEATURES=device-id-for-hardlinks=true might be an imperfect solution. But I don’t really understand it for now haha, so I would appreciate a quick rundown of any key drawbacks of enabling this.

• Furthermore, if LUKS full disk encryption is enabled, then a BTRFS snapshot is a snapshot of the decrypted disk information right? If I do a full system restore (e.g. doing a Restic restore at /mnt where the target disk is mounted, from within a live USB boot.), I would have to set up the full disk encryption over again. Is there a way to avoid this?

  Edit: Now that I think about it maybe this isn’t really a problem. I can just setup my target disk with a standard Fedora ISO and enable LUKS during the installation process. Then, I can use cryptsetup luksOpen (in a live USB boot) and restore from there. This should result in a restored disk with LUKS, right?

• Or would something like Clonezilla be better for a full system backup?

My end goal is to have a frequently backed up system, so I don’t lose any important data if any sudden and catestropic data loss occurs to me, and to make the backup easy to restore from if I’m panicking from sudden data loss.

[gurkan]

Hi

For a full system backup, IMO best would be to use some kind of snapshot-able filesystem / volume manager and do periodic snapshots, then backup these read-only snapshots via either restic or any other tool.

But honestly, if you’re not doing anything exotic with your system, backing up your /usr folder won’t help much, since that you’ll get as-is* when you do a fresh install of your OS anyway. If you really want to keep your setup I’d advise you to backup your configuration files, maybe home directory (which will be way easier with snapshots, but less pain if you just run restic towards it). If the target is to be able to restore whole OS as fast as possible (considering you’re not planning to use nixos or zfsbootmenu), full-disk cloning tools like Clonezilla would be the way.

>* “as-is” for any practical purpose

[Grass]

I see, thanks for your reply. I’m leaning more towards Restic backups of BTRFS snapshots than Clonezilla, because its more convenient to be able to just run a script/systemd service than to have to restart my system and boot into Clonezilla.

To that end, I’m deciding between using systemd and a shell script. The systemd approach allows for nice scheduled backups, but I’m worried that if it activates at an inopportune time, it might use up a lot of my system resources and cause it to lag undesirably. How significant of a concern is this? On the other hand, I can just backup at exactly whenever I want to by executing the shell script.

Lastly, is there anything I should know aboutRESTIC_FEATURES=device-id-for-hardlinks=true, other than it being some magic that fixes metadata issues with backups of system snapshots?

[quietlotte4878]

For the systemd-vs-script part, I’d still use a timer but make it gentle: set a time when the machine is usually idle, and run the backup with lower CPU/IO priority if needed. A Btrfs read-only snapshot plus restic is a good daily backup pattern, but I would also do one documented test restore to a spare disk or VM. That matters more than choosing the perfect scheduling method.

[Grass]

I see, thanks. Yeah I was planning on testing my backups maybe every month or so, on an old laptop

[Grass]

Question

What are the pros and cons of using BTRFS snapshots vs mounted LVM snapshots for Restic to backup from?

Context

I’m still in the middle of my research/reading, but my current understanding is as follows.

BTRFS

BTRFS snapshots have no recursive option. So, one would need to manually keep track of all subvolumes and merge them together. E.g.

$ btrfs subvolume snapshot / "$snapshot_path"
$ rmdir "$snapshot_path/home"
$ btrfs subvolume snapshot /home "$snapshot_path/home"
$ restic backup "$snapshot_path" ... 

I was thinking of making a script where I explicitly declare what subvolumes I want to merge (or not), amongst those listed in

$ btrfs snapshot list /

Here, I would use the results from

$ findmnt -t btrfs --pairs --shell

to obtain the correct paths, e.g. root → / and home → /home.

In other words:

• Advantage: No need to manually allocate storage for BTRFS snapshots, unlike LVM snapshots.
• Disadvantage: You need to manually make sure that you individually snapshot all subvolumes you want, and that you don’t forget any. If you ever add subvolumes and forget to add it to your backup script/service, you might face data loss when data recovery is needed. Alternatively, something similar to the above approach of using btrfs subvolume list and findmnt can add as a layer of checks. But this added complexity is another additional point of failure?

LVM (block level)

• Advantages:

  1. As long as you keep track of all paritions you wish to backup (easily tracked with lsblk), you’re safe. With BTRFS, you have to keep track of all partitions/BTRFS filesystems (e.g. /dev/sda3 on / and /dev/sda4 mounted at /home/user/share), and all subvolumes within each BTRFS partition.
  2. LVM is compatible with many filesystems, e.g. ext4 and btrfs. You can be reasonably certain that your backup strategy will work without changes if you ever decide to additionally backup a partition that’s formatted differently.

• Disadvantage: you have to manually allocate the amount of storage to allocate to the LVM snapshot. If the size of changes written to the source of the snapshot exceeds the storage you allocate, the snapshot is rendered invalid.

[shd2h]

@Grass

it doesn’t address the issue of ongoing filesystem writes during a backup, which I heard can cause issues with the backup?

Generally under linux this is not a problem, multiple applications can access the same file without issue. It is an issue under windows, but restic integrates VSS support to mitigate that.

The primary benefit snapshots provide under linux is point-in-time consistency of every file in the backup, which may, or may not, be important depending on how busy the system is in terms of writes, and how big the backup dataset is.

---

I can’t speak much to the LVM setup, but you appear to understand the btrfs snapshot setup well enough.
It might be simplest to just treat or think of btrfs subvolumes like they’re individual partitions, because in terms of snapshots they behave a lot like they are.
Which one you should use depends on how your machine is configured though surely? If you have a btrfs setup, you should use that. If you have ext4 or xfs, LVM is the only option.

If you want my advice? Keep it simple. If and when the unthinkable happens and you need to restore from backup, you want the process to be as quick and painless as possible.
i.e. use snapshots only if you actually need them, and take a full system backup only if you actually need to. Backups of user data are much simpler to work with.

Regarding btrfs snapshots/full system backups, you may find the discussion in this thread interesting/useful:

Bare-metal backup/restore with Restic?

I’m running Fedora 43 (KDE) and I want to use Restic to back up my workstation’s two drives. The drive that contains my /home folder (a 4TB SATA SSD) will be a simple file-backup to an external drive (and to Backblaze B2), and I’m pretty sure I understand how to set that one up. The main boot drive, though (a 2TB NVMe SSD) – that, I’m not quite certain how to do yet. I have Snapper installed on my machine, and it makes regular snapshots of that boot-drive (yes, it’s BTRFS – both my drives are…

Whatever approach you decide on, please make sure you thoroughly test the restore process you come up with. Backups are worthless if they cannot be restored, and a full system restore is always going to be significantly more complex than a regular “user data” restore, so you need to be sure it works before you have to rely on it!

[Framsfex]

You could use fsfreeze to stop writes to the filesystem while restic is running or use snaprotate with option -r (recursive), restix offers both options.

[Grass]

Oh, thanks for introducing me to fsfreeze and chattr. I don’t think I can use fsfreeze on the whole filesystem / without issue, so I’ll probably stick to BTRFS snapshots. But it seems like quite handy. So does chattr, I think it’ll be helpful in managing my .secrets directory.

For now I wrote a simple shell script to Restic backup a combined BTRFS snapshot. But I’m planning to fork btrbk where I’ll add the ability to declaratively declare how to combine BTRFS snapshots into one and back it up with Restic, configured in the same file. Maybe I’ll also add the ability to do LVM snapshots and back them up, and fsfreeze in case neither BTRFS nor LVM is available.

I’ll probably try to enhance security by exploring Linux capabilities, securebits, Landlock, etc, just for fun. And I’ll implement an optional strict mode to ensure all subvolumes are explicitly configured to be snapshotted or not, and to be Restic backed up or not.

Finally, perhaps I can write a script to easily restore a filesystem snapshot via live USB to another device, which allows quick system restoration in case of panic.

Well, I don’t know how much I’ll be able to do or how well I’ll be able to do it. But it seems interesting haha