I’m using restic 0.9.4 on Centos 7.x to back up a nontrivial (several terabytes) amount of data to B2. The process is started via cron, and is meant to take a snapshot of our NFS share every night at midnight.
Our data, typically falls into two categories – user created and therefore “precious”, and machine generated and therefore “trivial”.
for simplicity, all the ‘trivial’ data lives in a folder we call ‘footage’. Typically, the data footprint in the footage folder (which we don’t want to back up) overwhelms all the other data. Here’s an example:
[gene@tws09 cmn]$ sudo du -sh /projects/my_project 2.2T /projects/my_project [gene@tws09 cmn] cd /projects/my_project [gene@tws09 cmn]$ sudo du -sh * 7.3G assets 3.0M deliverables 784M edit 2.2T footage 68K onset 68K supers
As you can see, the amount of data i’m interested in backing up on this project, adds up to approximately ~8.3G [as the 2.2T footage folder is being skipped].
now for the fun part:
[gene@tws09 pxx_010]$ ps -ef | grep restic root 24431 95350 33 Mar23 ? 07:05:43 restic --exclude "**/footage/**" -r b2:my-bucket-2019 backup /projects/my_project
As you can see, this process has been running for over 7 hours.
Our internal network is 10Gbe, and we have a 1Gbe fiber internet uplink, which we know to be reliable, and while it’s true that b2 is not the fastest platform, I suspect that something else might be going on here.
Now, for the odd part, looking at the files that open/being accessed, I noticed that restic seems to be processing files that I’m explicitly asking it to ignore.
[gene@tws09 pxx_010]$ sudo lsof -c restic restic 24431 root 24r REG 0,42 27344764928 266637223/projects/my_project/cmn/footage/raw/my_giang_27G_file.
Am I using the exclude flag correctly? should restic be looking at this giant ‘trivial’ file under the footage folder?
If anyone has any insight, it would be greatly appreciated.