Hi, i want to backup my whole linux system from my laptop, with excluding the windows install (i barely use it), but including the boot partition too, but if i’m not using switch --one-file-system restic also tries to backup system folders meta files, in /run or /proc, etc. This leads to an apparently infinite cycle ( stuck at counting the size of /proc/1/task/1/pagemap forever, showing larger backup than the actual size of the whole disk… ).
I need to add these exclusions one by one? Also for cache folders? There a lot of cache folder types, sometimes its .cache, cometimes its just cache, etc…
Are there any maintained exclusion lists to solve these? BTW this should be in restic i think, as these are quite common, to exclude all things that should not be backed up both from linux, and windows.
There’s some material about this if you use the search function in this forum with the keywords “full system”. The list of paths you need to exclude isn’t that large.
Regarding caches, you can use --exclude-caches to automatically exclude contents of directories that have been conventionally marked as caches.
General advice; Don’t try to overdo this The world doesn’t end if you were to back up some stuff that you didn’t really intend to, so exclude what you can think of and go for it.
tried these excluded from --exclude /dev/* --exclude /lost+found/* --exclude /media/* --exclude /mnt/* --exclude /proc/* --exclude /sys/* --exclude /tmp/* --exclude /var/cache/apt/* from post Lack of Documentation on how to do a full system back up
and now i see
lock repository
no parent snapshot found, will read all files
Turns out the cause is --exclude /dev/*, if i don’t have it then using parent snapshot why?
after some testing, i see if i use --iexclude={/dev/**,/media/**,/mnt/**,/proc/**,.......} then restic finds parent snapshot, but if i use the same paths in multiple iexcludes, then no parent snapshot. Like this --iexclude /dev/** --iexclude /proc/** and so on
using the same exclusion rules, on another linux machine, excluding all mounted drives, restic stuck after backing up 15 terabytes… from an 512 gbyte SSD… What is this?
Using the same command line, i see progress info and current backup info, on the other machine, nothing, just “backup started [/]”
I’m guessing there is some path which is a never ending story. Maybe you can use the verbose options of the backup command to increase verbosity to get the list of files restic tries to back up and thereby see what might be the culprit?
I think that this approach is never really going to work 100% trouble free.
And it is not restic limitation per se - simply IMO you are trying the wrong tool for the job.
You wan full disk backup? You should investigate how to create full disk image.
You want files based backup? Instead of hunting exclusions (you might have it right one day and it will break next day) ask yourself which data is important and used inclusions for your backup.
Or use modern filesystem like BTRFS or ZFS - take full disk snapshot and backup snapshot - this would be 100% safe and probably the best approach.
This. Back up your data. The times in your life that you have to restore your entire system from backup are so few that it’s not something to complicate things over. IF you truly want to be able to restore the entire system, I’d reach that goal by other means than a file-based backup tool (e.g. using NixOS or Ansible or whatever to make the system reproducible).
I’m not sure what you mean by seeing “much more things”.
My point is that if you use restic backup -vv --dry-run then you should see exactly which paths restic would back up. If you get different results between these two machines, presumably they either provide different sets of paths or one or more of the paths in the long-running system is providing a never ending stream of data. Regardless, inspecting the difference might yield some info. Perhaps you did, it’s unclear to me what you actually did to investigate this.
It’s also more useful if you provide the exact commands you run, and preferrably their output, when you have the problem and not, otherwise there’s too much potential ambiguity involved in trying to understand the exact details of the situation.
i cant access the old command anymore, as there was a power failure, and .bash_history file is lost.
But now copying from the other machine, i see progress info:
Using -vv writes out every backed up files, which is a huge amount. But just the unusually big files should be listed there, else i can’t find the problem. BTW are the rules from iexclude-file, and plain iexclude, excludes, etc combined when running a backup command? Or maybe the last/first is used only?
PS: it seems i’ve found it: /var/docker/plugins/uuuid/propagated-mount/
restic combines all excludes. Usually, the easiest way is to write all files into an exclude file and then use --exclude-file excludes. That avoids problems with shell escaping.
/dev/** and /dev are actually equivalent. There shouldn’t be much of a performance impact with recent restic versions, but it’s just unnecessarily verbose.
The world doesn’t end if you were to back up some stuff that you didn’t really intend to, so exclude what you can think of and go for it.