Since looking at
restic I have had the same thoughts as @alphapapa and @arikb
My performance problem with
restic is not the actually backing up of files or dedup or (lack of) compression, but the fact it spends 99% of its time pointlessly scanning 100,000’s of files to notice they haven’t changed since an hour ago That’s a lot of wasted effort and I/O that continuous solutions like CrashPlan and Carbonite avoid.
restic support backing up a list of specific files (
--file-from). So my thought was to have a
inotify wrapper/daemon that would build a list of modified files for 15-60 minutes, and the dispatch a
restic backup with the accumulated file path list. Then once every 12-24 hours it would run restic with a full scan to ensure nothing is missed (inotify and NTFS events are not guaranteed, they use kernel memory and just get dropped if not picked up in time).
A continuous wrapper would reduce hourly
restic backup times from ~30 minutes runtime to ~3 minutes. An greatly reduce the hourly I/O operations on the filesystem.
Has this been done already? Anyone seen a generic
inotify wrapper that could do this or be adapted to do this?
I guess a shell script with
inotifywait could do it for local filesystems.