I need to keep and empty directory tree for some path, how can I do that with exclude list, is it possible ?
A good example is done with rsync at http://psung.blogspot.com/2008/05/copying-directory-trees-with-rsync.html
Thanks
I’m not entirely sure what you want to achieve. If you want to archive a directory like /home/user/foo without its contents, use --exclude "/home/user/foo/*" (mind the quotes so that the shell won’t expand the pattern). Restic’s include/exclude rules are rather basic at the moment.
Can you give us an example what you’re trying to do?
The problem is if I --exclude “/home/user/foo/*”
I also exclude all directories inside foo (no ?),
what I would like is to be able to precise that I want to keep
-/home/user/foo/images
-/home/user/foo/videos
-/home/user/foo/music
without any file in any level of the hierarchy.
I would like to keep only the directory tree for these directories.
I think this can be achieved with a workaround. Copy the folder structure to a special directory and backup that directory.
See this link for examples to do this: how to copy the directory structure without the files
Thanks, your answer at least it tells me I didn’t miss an option of restic which makes it an easy play.
If I have to script it I can also, for example, do a find -type d before backup put the result in a file and, when time of restore comes, don’t forget to do mkdir on each line of the file.
But that’s not elegant, as my aim is to be able to restore without thinking about what will or will not work. Imagine the restore comes a few years after the backup and I completely forgot about this file or about your deported directory tree that has to be moved back on its right place…
Perhaps it’s possible using a combination of excludes and includes, but if the tree is deep or has many dirs, that makes it also a pain in the a…
Wouldn’t it be a nice thing to have the option in restic options ? I mean a --only-dirs option for example…
That’s correct. At the moment, you can’t do that with restic…
While it would be nice (for you in particular) to have this feature, I doubt that it’s so useful for the vast majority of users. So I must say I’m not convinced that it’s a good idea to add this. Sorry about that 
Here’s a nice article explaining the options dilemma: Tech Notes: Why not add an option for that?
…For me and all Mac users.
In fact I’m just trying to mimic TimeMachine using Restic, and TimeMachine has an exclude list made of 4 types of excludes, here are the comments from the Apple file
paths we do not want to include in a system backup
(ex : /.Trashes)
paths where we need to capture top level folder to restore disk structure, but don’t want to backup any contents
(ex : /private/tmp/*)
standard user paths we want to skip for each user (subpath relative to root of home directory)
(ex : /Users/*/Library/Caches)
and finally
- paths where we need to capture entire subtree folder layout to restore disk structure, but don’t want to backup contained files
The Apple StdExclusions.plist lists
/private/var/log
/private/var/spool/cups
/private/var/spool/fax
/private/var/spool/uucp
If I understand the Apple comments, this is needed if I restore an OS X system, so that the system works.
That’s why I’m looking for this option.
This won’t preserve permissions, which will likely be a problem if those directories are used by an operating system.
You are right, that’s not elegant, but it works. You could save a text file with instructions to restore the directories so you don’t need to remember every detail.
To be honest, this is the first time I’ve heard about a backup software having this option.
As a Mac user, I can say I have no need for backing up empty folders. If it’s empty, I really don’t see a need to back it up. (That’s just me, personally.)
You don’t but if you use TimeMachine you do, without knowing it.
That’s one of the things that makes a TimeMachine restore running smoothly 
How does restic exclude list handles spaces in filenames ?
Do we have to escape spaces
/this\ is\ a\ filename.txt
or to quote it
“/this is a filename.txt”
in exclude file list ?
IBM Tivoli has this feature too, exactly for this reason: sometimes you need to keep the directory structure (e.g. for /var/cache/) but not the files contained.
You just write the file names into the exclude file, without treating spaces differently (no quoting, no escaping).
There’s another similar use-case on Linux webservers.
If we restore a complete apache-based webserver with the logs directory omitted (/var/log) then Apache doesn’t want to start because we have per vhost log files in separate directories: /var/log/apache2/site1/access.log, /var/log/apache2/site2/access.log. Apache is unable to create the logfiles after restore as the site1 and site2 folders do not exist.
We recreate the directory structure after restoring a backup.
Glad to see I’m not a fool with my empty directories 
I ended with building the exclude file before backup each time with
$ find /private/var/log -type f > ExcludeList.txt
But that’s not really elegant, no ?
I’m not sure it’s a feature that’s worth implementing. Let’s do a poll: How many people here need (like, really need) an option to only backup the directory structure within a certain path? Please like this post
In terms of the user interface, I can imaging an option --exclude-files-in /var/log (name to be discussed) so that everything that’s not a directory is excluded, but only within /var/log.
3 Likes
fd0, I’m aware you already have a lot of work on Restic, but asking for a poll seems to be a way to put the option away forever.
Why not give it a very low priority in your todo list instead ?
The need exists on OS X, according to sorcix it exists for Apache (which can be on a bunch of OS)
and it exists for IBM Tivoli (according to rfc1036).
That means it’s not a crazy option from a lonely fool with a very unique system,
these three examples are clues that the option could be necessary on many other cases we’re not even aware of. They’re only the top of the iceberg.
Looking deeper it brings us to the restore strategy.
At restore time, do we hope for a running restore or do we plan to reinstall system and apps from scratch and then get our personal data from our Restic backup ?
If hope is getting things running smoothly from a restore then the option makes sens,
if we plan to redo installations, an OS X install or an Apache install would surely rebuild the directory structure.
As I was trying to mimic TimeMachine (the Apple backup process) which is a restore and forget system, I discovered that these empty directories were necessary.
I don’t know if the lack of these directories would avoid booting OS X or only feed the logs with errors.
In my experience Apache is very sensitive to errors and I’m quite sure the lack of directories would prevent apache from running.
If you don’t see this option as essential it’s ok but don"t throw it away forever, some of us have hope to see it someday.
For now I can circumvent the lack by building a big exclude list containing all the files to avoid in these directories.
Thanks for reading and thanks for this wonderfull piece of software.
V.
Interesting point, thanks for sharing it. I’m not yet convinced this is something restic needs (there’s the options dilemma, see the article I linked above), but I’m also not saying we’ll never implement it. The described use cases are valid in my opinion, and I didn’t see them before, so your explanations were worthwhile.
While the workaround with a generated exclude list containing all files works for now, I agree that it’s technically rather ugly. I could be improved a bit by using shell magic (restic backup --exclude-file <(find /var/log -type f) /var) but that doesn’t make it much better.
From a technical standpoint, the exclude logic is already extensible enough to add this. We’d also need find a good user interface. Excluding things turned out to be very tricky, almost as complicated as applying forget policies 
So, if you’d like to see this feature eventually (or an official rejection, should we decide to not implement it), please create an issue on GitHub so we can track its status. No need to fill out the issue template (which is mainly meant for bug reports), just add a rough description so I can remember what the feature is when reading the ticket later. Thanks!
1 Like