Unable to exclude FUSE mounted directory

Getting Help
1

I started playing with Keybase on my Linux machine. The default install creates a FUSE mounted /keybase directory at the filesystem root. This directory can only be read by the user that mounted it and not the ‘root’ user. And I am unable to exclude that directory from my backups.

Here is my nightly backup run manually

$ sudo restic backup --one-file-system --exclude-caches --exclude-file exclude.list  /
[sudo] password for wscott:
password is correct
using parent snapshot 4c575134
scan [/]
error for /keybase: lstat /keybase: permission denied
scanned 22135 directories, 187197 files in 0:10
error walking dir keybase: Lstat: lstat /keybase: permission denied2 items  0 errors  ETA 1:00
[0:41] 100.00%  212.232 MiB/s  8.594 GiB / 8.594 GiB  209331 / 209332 items  1 errors  ETA 0:00
duration: 0:41, 211.55MiB/s
snapshot 0bba673e saved

The exclude.list file includes '/keybase', but the restic tries to look at it anyway.

Here you can see how that directory behaves. Notice that the directory appears to be missing .. and ..

$ ls -la /keybase
total 0K
dr-x------ 1 wscott root 0 Nov  9 13:44 private
dr-x------ 1 wscott root 0 Nov  9 13:44 public
dr-x------ 1 wscott root 0 Nov  9 13:44 team
$ ls -ld /keybase
dr-x------ 1 root root 0 Nov  9 13:44 /keybase
$ sudo ls -ld /keybase
ls: cannot access '/keybase': Permission denied
$ sudo ls -l /
ls: cannot access '/keybase': Permission denied
total 2097305
drwxr-xr-x   2 root   root      12288 Oct 27 06:38 bin
drwxr-xr-x   4 root   root       4096 Oct 27 06:39 boot
drwxrwxrwt  13 root   root       4096 Oct  5 11:14 build
drwxr-xr-x   2 root   root       4096 Sep 17 05:23 cdrom
drwxr-xr-x  22 root   root       4880 Nov 10 18:08 dev
drwxr-xr-x 159 root   root      12288 Nov 10 21:27 etc
drwxr-xr-x   4 root   root       4096 Oct  3 15:39 home
lrwxrwxrwx   1 root   root         33 Oct 19 11:15 initrd.img -> boot/initrd.img-4.13.0-16-generic
d?????????   ? ?      ?             ?            ? keybase
drwxr-xr-x  23 root   root       4096 Oct 25 09:37 lib

Suggestions on how to prevent restic from trying to ‘stat’ an excluded directory?

2

Restic will stat() all files/dirs before the exclude filter is considered. That’s necessary for example to find out if a file system is mounted at a directory, then the os.FileInfo will contain a different device ID.

We could run name-based excludes before doing a stat, but that will complicate the code. Or we could ignore errors from files/dirs that are to be excluded anyway.

3

Notice that the error happens twice. The file scanning path and in the backup path. Is it really traversing the directory tree twice?

When writing code like this I often need to put tests for excluding directories earlier in the pipeline than the test for excluding files. But rerunning the exclude code in the error path to suppress an error like this seems reasonable too.

4

I encounter the same issue here with a directory /home/xxx/A which I exclude. However, in my case I only get an error while scanning. The directory /home/xxx/A below seems to get excluded correctly in the backup process. By the way I am using the master branch from git hub.

using parent snapshot d05799e5
scan [/]
error for /home/xxx/A: lstat /home/xxx/A: permission denied
[0:52] 75314 directories, 636080 files, 35.715 GiB
scanned 75314 directories, 636080 files in 0:52
[2:24] 100.00% 0B/s 35.715 GiB / 35.715 GiB 711393 / 711394 items 0 errors ETA 0:00

5

Yes, it is. Otherwise it’d be impossible to inform the user how far along in the process we are. There’s an issue which tracks not doing the scan if that’s not necessary, #1160