Restoring files directly into a single compressed archive

#1

Is there any discussion about a way to restore files to a single .zip or .tar.gz file?

Am willing to write code. :slight_smile:

I could envision a couple ways of this happening:

  • Direct support implemented into restic’s CLI, with a flag such as --archive zip or --archive tgz or something.
  • The restore functions exposed as a library so that an io.Reader can be returned such that I can pipe the contents of the restore into a zipper or targz-er function. Ideally avoiding writing a whole archive to disk at once.

The goal is to stream the contents of the restore out to some client over a network, while preserving disk space and memory.

This is a pretty niche need, so, I am of course willing to write code, even if it’s external to restic if it wouldn’t be a good fit for restic itself.

What do you think?

Snapshotting a directory from STDIN?
#2

Hi @matt you can already achieve this in at least two ways:

  1. Mount the restic repo at a file system with restic mount and then zip up the snapshot folder you want, with any filters you want, to a zip file.

  2. Mount a zip file as a fuse filesystem and then restore with restic into that flle system. Unmount the filesystem when finished and you have the zip file. (https://bitbucket.org/agalanin/fuse-zip)

Both method avoid ever restoring all the files to a real local file system that takes up space. In both cases the files go straight to a zip.

If you want to stream the zip file directly out of restic to e.g. some sftp location, then you will still need to cut some code I think.

#3

I don’t know much but maybe at some point the restored snapshot or file would be in a tmp folder in order to create the compressed file and then it’ll need to delete the restored files leaving just the compressed one in order to accomplish this. I think using mount option and tar is the simplest way. Once the directory is mounted you just need to point tar to the snapshot you want to save.

#4

Thanks both.

This is an interesting possibility. I don’t love the idea of trusting a fuse mount since they tend to be unreliable and/or slow, but I’ll look into how viable this is for our needs. Still preferable would be for restic to have some native support for this, rather than relying on a fake file system, but if it does the job (even if it’s a little slow), I can work with that.

Thanks for the idea!

#5

Correct me if I am wrong

Restic mount doesn’t work on windows…

1 Like
#6

I don’t think WSL supports fuse yet. You can vote for the feature here:

You can use a Linux VM or Linux docker container on windows and do it in there, with the output ZIP going to a mounted Windows filesystem.

#7

Instead of implementing a compression directly into restic, why not simply support a tar output stream, so it can be piped to other compressors on the shell?

2 Likes
Snapshotting a directory from STDIN?
#8

@lukastribus this is already supported, see the --stdin option. You can backup a compressed stream, using your own compressor of choice.

If you backup a compressed stream you can restore it as a compressed stream. That wasn’t the problem. @matt would like to backup up uncompressed files and then restore the files and compress them without first writing them to disk.

#9

@whereisaaron Your reply to @lukastribus misunderstands him.

He is talking about restoration, not backup, so --stdin is not relevant here.

#10

And actually I already got this working for my own needs on Linux. (It’ll power Relica’s web-based restore.)

#11

@cdhowie @matt if you don’t mind restoring the whole archive file, then you can just restic dump snapshotID file to get it on stdout, and pipe it to your compressor.

The “dump” command extracts a single file from a snapshot from the repository and
prints its contents to stdout.

#12

I’m not sure how that’s relevant to the thread. This thread is about having restic restore multiple files from a snapshot into an archive format, not as individual files to disk.

restic dump only pulls a single file from a snapshot.

#13

Yes it does work, you run a dump for each file as you add and stream them to the same output archive.

That does require a script or small code wrapper. My original suggestion (above) was restic mount, since you can point your regular archiver directly at the mount and use its normal file selection. Or use the archiver’s own mount feature to restic restore --target directly into the archive.

#14

Sure. We can also do everything on our computers by writing in assembly language.

There’s nothing restic does that we can’t do with scripts, either, so why do we need restic? Just use scripts!

I understand your point, but consider mine: this is a pretty general feature that would be rather widely useful. Having each user re-invent their own scripts is a waste of collective time.

Additionally, using restic dump multiple times from a script requires a restic invocation for each file, which is not cheap at all. Restic has to find the right key, decrypt it, acquire a repository lock, open the desired snapshot, descend through the tree structure to find the file, output the file, then clean up the lock. Doing all of this for each file would slow the operation down tremendously. For more than a few hundred files, this approach is completely infeasible.

This is a sane suggestion, though it only works on platforms were fuse is available. A proper “dump some files as an archive” command could be useful on platforms where fuse isn’t available.