Reliably creating a tar export without jumping through hoops

Getting Help
1

Hi there, new user on the latest version on Windows. Enjoying restic so far, my complements!

I am using restic in the excellent git for Windows bash terminal.

$ restic version
restic 0.17.1 compiled with go1.23.1 on windows/amd64

My use case is that I want to export tar archives of entire snapshots from various restic repositories. This is useful for existing tooling that confirms the validity of our disaster recovery efforts. So far, I cannot find a way to reliably automate this without first investigating each snapshot.

As an example, this particular repository backs up a saver. The full Windows path is some variant of C:\Users\<USER>\AppData\<...ETC>...\Data, but full paths are not useful to me so I backup from a relative folder path. The ls command demonstrates this:

$ restic ls latest
# ...
/Data
/Data/<...VARIOUS FILES...>

Despite restic ls clearly showing that all paths start with /Data, I was not able to run restic dump on this path:

$ restic dump latest '/Data' --archive tar --target export.tar
# ...
Fatal: cannot dump file: path "\\C:" not found in snapshot

I expected restic ls paths to work in restic dump, hence my confusion.

Incidentally, this failed command creates a zero-byte export.tar. You might want to consider that behavior a low-priority bug, considering that it encounters a fatal error before any files are exported? Up to you.

Try as I might, I cannot get restic to “please just export the whole damn snapshot”:

# Same error as above
$ restic dump latest '/' --archive tar --target export.tar
# ...
Fatal: cannot dump file: path "\\C:" not found in snapshot

# Tried using a windows slash instead, slightly different error
$ restic dump latest '\' --archive tar --target export.tar
# ..
Fatal: cannot dump file: path "\\" not found in snapshot

# Tried using a dot
$ restic dump latest '.' --archive tar --target export.tar
# ...
Fatal: cannot dump file: path "\\" not found in snapshot

Eventually I found what it wanted: Data without a leading slash works!

$ restic dump latest 'Data' --archive tar --target export.tar

# Worked!
$ tar tf export.tar
Data # ...

One small surprise I had is that restic dump will happily clobber existing files. Considering how other parts of restic are quite cautious, I found that a little incongruous:

$ touch some-file

# Happily clobbers file without confirmation
$ restic dump latest 'Data' --archive tar --target some-file

For now, my workaround to reliably restic dump a whole snapshot is:

  1. Investigate the output of restic ls
  2. Check if there is a single top-level folder, or multiple
  3. Remove leading slashes
  4. Pass those results to restic dump

I could script that with restic’s --json flag, but first I wanted to check with this forum that my understanding is correct & that I’m not missing something obvious.

My questions for this thread are:

  1. Is the mismatch between paths used by restic ls and restic dump intentional? How can I understand the difference?

  2. Is there a way to run restic dump reliably on a whole snapshot without looking at things like restic ls?

  3. What is the official restic policy or design on slashes differing between OSes? Does restic consider everything to be POSIX internally? Is this documented?

  4. Do you (the devs) want to consider initial restic dump --target failures creating a zero-byte tar a bug?

  5. Do you (the devs) want to consider restic dump --target overwriting files without confirmation a bug? Or, maybe document that behavior in --help?

Thanks!

2

See Not parsing correctly directory path in Windows 10 using Git Bash · Issue #5069 · restic/restic · GitHub .

tl;dr git for windows messes up the file paths (well, translates them to windows path. Which happens to be wrong here) before they are passed to restic. You’ll need to find a way to prevent that from happening. Maybe there’s also an official syntax to do so.

3

Both commands use the same path structure. This is a git for windows problem.

restic dump latest / does exactly that.

The repository format does not know anything about filepath separators. There’s no real policy, but all core developers don’t use Windows as their primary OS. So, there’s little motivation to use anything other than posix filepaths. Those are also arguably less messy to work with than Windows filepaths.

The file separator handling in a few places in restic is unfortunately somewhat weird at the moment (unrelated to your issue). I don’t recall that there’s any explicit documentation on how Windows filepaths are handled.

Hmm, we can change that if someone creates a corresponding PR. So feel free to open an issue, but it will probably only be fixed if someone steps up and makes the required changes.

That is expected behavior. There’s also no such prompt in other parts of restic that write files.

4

I appreciate the detailed reply!

From the #5069 bug you linked, I can verify that the environment translating paths is the problem:

$ DEBUG_LOG=restic.log restic dump latest '/'
# ...
Fatal: cannot dump file: path "\\C:" not found in snapshot

# First line of restic.log
main []string{"C:\\...\\restic.exe", "dump", "latest", "C:/Program Files/Git/"}

Switching to '//' as you suggested in the bug report fixes it:

$ DEBUG_LOG=restic.log restic dump latest '//' 
# Works!

# Arguments passed to restic are now correct
main []string{"C:\\...\\restic.exe", "dump", "latest", "/"}

Should have thought of that… Have a good one :grin:

1 Like