Backup Misinterprets All Directories as Modified

Getting Help
1

Hello,

I’m encountering an issue while creating backups with Restic and would appreciate some assistance.

Here’s my situation: I have data in a specific folder on my disk that can change and even be completely deleted (unmounted) by users. I need to back up this data roughly once a day. It is crucial that the files do not change or get deleted during the backup process.

To handle this, I decided to copy the files to a temporary folder and create the backup from there. When copying the files, I use the cp -rp command to preserve all attributes and permissions. However, when I create the backup with Restic, even though I specify the correct parent snapshot ID, it still thinks that all the directories have changed, even though they haven’t.

Here’s my process:

  1. Copying files from the source folder to the temporary folder: cp -rp /source_directory/* /temporary_directory/

  2. Creating the backup from the temporary folder using Rustic and specifying the parent snapshot ID. sudo restic -r /mnt/user-data/777/u-7777 --password-file /root/.restic_password_file.txt backup --host worker --parent 9c12c1d9 --ignore-inode .

As a result, Rustic considers all directories to be modified, leading to unnecessary full scan, which is inefficient.

repository b9e19c33 opened successfully, password is correct
using parent snapshot 9c12c1d9

Files:           0 new,     3 changed, 117948 unmodified
Dirs:            0 new, 27983 changed,     0 unmodified
Added to the repo: 51.186 MiB

processed 117951 files, 2.758 GiB in 0:04
snapshot e322d117 saved

Can you please advise what I might be doing wrong or suggest any additional options I can use to avoid this issue?

Thank you in advance for your help!

2

Welcome @bushev !

Then you need to use the snapshot feature of for example BTRFS or LVM and backup that instead of backing up a copy. Or use rsync to keep a single copy up-to-date each day instead of a new copy every day. Read on…

That is because all files and directories have new inode numbers, because you are making a new copy each day.

1 Like
3

Just asking: is it worth doing the temporary copy? Couldn’t it be that the files are in an inconsistent state during the copy just as during the restic backup? I would guess that copying the files takes longer than restic and has more chances to read opened files. But I could be mistaken.

2 Likes
4

I’ll agree with what others have already said; snapshots of the backup target are (IMO) the best solution to ensuring all the files are in a “consistent” state during the backup.

Something that stood out to me in particular though:

can change and even be completely deleted (unmounted) by users

If the target directory being unmounted is a particular issue, you might want to consider making the restic job dependant on the target directory in question being a mount point.
If you’re triggering the restic backup via a systemd service, that’s fairly straightforward, see the “ConditionPathIsMountPoint=” systemd unit condition (documentation link).

1 Like
5

I attempted to clone a volume and use it as a backup source, but unfortunately, the outcome remained the same - all directories are marked as modified. Here’s what I did:

sudo lvcreate -n snap-u-7777 -s my-vg/u-7777
sudo lvchange -ay -Ky my-vg/snap-u-7777
sudo lvcreate -y -V 8G -T my-vg/data-pool -n tmp-u-7777
sudo dd if=/dev/my-vg/snap-u-7777 of=/dev/my-vg/tmp-u-7777 bs=4M
sudo mount /dev/my-vg/snap-u-7777 /mnt/snap
sudo e2fsck -f /dev/my-vg/tmp-u-7777
sudo tune2fs -U random /dev/my-vg/tmp-u-7777

Next, I proceeded to run the backup command:

sudo restic -r /mnt/user-data/777/u-7777 --password-file /root/.restic_password_file.txt backup --parent be4b49fe --ignore-inode .

repository b9e19c33 opened successfully, password is correct
using parent snapshot be4b49fe

Files:           3 new,     0 changed, 117948 unmodified
Dirs:            4 new, 27979 changed,     0 unmodified
Added to the repo: 50.059 MiB

processed 117951 files, 2.758 GiB in 0:04
snapshot 717fbaf6 saved

Bear in mind, I used --ignore-inode to avoid any inode issues. But I’m still facing some problems. So, what could be the potential cause?

6

With LVM snapshots you’re probably affected by this issue: Restic 0.10.0 always reports all directories "changed", adds duplicate metadata, when run on ZFS snapshots · Issue #3041 · restic/restic · GitHub . restic 0.17.0 will contain an optional workaround, but finding a permanent solution to prevent the changed directories issue is still in progress.

7

Yeah, that explains!