Disclaimer:
I am new to restic and, even though I use Linux (Debian Testing) almost daily since around a year, I am not too familiar with all its internal things until now. So please excuse if some of the following questions seem to be irrelevant or if the answers seem are evident. I will do my very best to provide all necessary information.
My System:
I have a dual boot system with Windows and Debian. Windows is running on an internal SSD (sda1, probably NTFS). Debian is running on an internal HDD (sdb1, ext4, has a small, drive internal SSD Caching). Both OS share a “data-partition” on another internal HDD (sdc1). df -T tells me its fuseblk, which seems consistent to me, as I mount it using the /etc/fstab file. I started using this drive/partition, when I was only using Windows. For that reason it is ntfs. It is mounted under /home/myUsername/Documents.
What I aim to do:
Because all of my important Data is on this data-partition and almost only there, I definitely need a up-to-date backup of it. I want to write this to an USB plugged external HDD drive (at least until I have a better (remote) storage available).
I don’t care for either sda1 nor sdb1, where only my systems and programs are running on. In the worst case I’d loose some configurations, which is acceptable.
Questions:
Is it at all possible to backup the mounted ntfs/fuseblk drive to the external USB drive using restic? I assume yes.
While reading restics documentation I found this hint in the section about the “File change detection” (under “Backing up”):
“The option --ignore-inode exists to support FUSE-based filesystems and pCloud, which do not assign stable inodes to files.”
But I am not sure if I get it:
2.1) Thats ment for my case, right?
2.2) If I don’t use it, restic will probably rescan all (or at least some) files on that bulk at each backup, right?
2.3) Using it brings the risk of missing some information when backing up, for example in case I change rights (chmod) for some files?
In my mind everything else should work “perfect”. Is there anything I have to care about or that comes to your mind, what I might not have thought of?
Edit:
I want to name you the commands I plan to use:
I’ll try to answer, but take with a grain of salt since I didn’t need to touch ntfs for a long time. Corrections welcome.
Yes, as long as restic has read access to the source and write access to the repo on the external drive.
< placeholder for formatting>
2.1 Yes, afaik FUSE does not provide reliable inodes.
2.2. Sounds like all files will be considered changed on every backup. Deduplication would still recover a lot, but it’d be still unnecessary.
2.3. As far as I can remember, ownership on ntfs mounts was decided by mount options, so I don’t think you will lose something with that option.
Sounds ok, don’t forget the option mentioned on 2nd point on backup command.
@gurkan many thanks for your reply!!!
Nice to hear, that (at least) most of my assumptions seem to be correct.
I made my first backup during this night and it seems everything worked as planed. I did a second backup right after that, to check if the rescan with --ignore-inode flag works properly. I’m not sure, but I think it did. It took only 1:06 minutes instead of 6:30 hours. There where 3 changed files and 6 changed dirs (as per restic output). Those can be explained by the work of an Owncloud client working in a small subset of my data-partition.
Just for curiosity I did another backup just now, to see what happens, if the PC was restarted in the meantime. This one took 1:25 minutes, so it was also very fast. Unfortunately restic says there where all 27140 dirs changed, which means it adds around 90 MiB to the repo. That can definitely not be explained by that Owncloud client. My knowledge is insufficient to explain that. Does anyone have an explanation? Is that something I have to deal with (maybe because of ntfs) or can I do anything against it?
In my proposed weekly backup cycle that would be 5 GiB a year, without any change in data. But it would be much more in case of a daily backup…
One general request, because @gurkan is not absolutely sure about his answers: Can anyone confirm his answers (or comment on them)? If its confirmed, I will mark them as solution…
You can use restic diff $snapshot1id $snapshot2id to see what changed. It will show the reason to some extent (you can check restic diff -h for it). If the metadata was reason, you can retry with --ignore-ctime maybe. Since it’s fuse, I can’t theorize what’s going on.
The output of diff --metadata shows a change of metadata for every file (at least it looks like it is every file). So yes, that seems to be the reason. Normal diff shows only four changed files (as expected).
I can not explain in how far the change in metadata of the files itself results in restic naming changed dirs, but maybe I just misunderstand restics output?
Anyway I wonder if --ignore-ctime can help, because according to the documentation--ignore-inode should already “allow ctime to differ”.
But it is possible I just did a mistake, when specifying the flag. In the first version of my command I received an error. Therefore I moved that flag to the end of my command. This was it: restic -v -r /media/myUsername/INTENSO/Daten/restic-repo_data-partition backup /home/myUsername/Documents --ignore-inode
Probably the --ignore-inode must be specified directly after the backup subcommand?!
This shouldn’t be an issue, I mostly give flags after the target and it never failed to recognize.
Normally when I mount the repository with mount option, I can inspect the differing files via stat command and see most of the important stuff. Could you check a changed file within different snapshot to see what is specifically changed that way?
Hey @gurkan, as you will have noticed (for sure), I did not respond to your last Idea/Question. That was neither because the problem was gone, nor because I’m not interested to try this. It was just for the reason of easter time and several urgent tasks in the following weeks. So please excuse my missing response.
Anyhow I tried to follow your advice to get some data about my problems.
As far as I can see both stat outputs (one of older snapshot and of the one with changed dirs) only differ in the inode number. I can not interpret that, but I reflect if that is the “original” inode number or if its the “virtual” one of the repos fuse mount (which would be irrelevant)?
Until now I only tried the basic stat command without any options. If I should try anything else, please specify this. And by the way, I made sure to use an “example” file that was named as changed by restic diff --metadata. Kind regards!
I had to re-read the thread to remember
I am not sure at this point. One last idea: Do you always mount the source to the same place before backup? Also not sure if it is about parent, but it might worth a try to use --parent while using the backup command and giving the previous snapshot as a forced parent, if for some reason it can’t determine from a path change etc.
There where 3 changed files and 6 changed dirs (as per restic output). Those can be explained by the work of an Owncloud client working in a small subset of my data-partition.