Downloading a snapshot "patch"

Let’s say I have two snapshots:

ID        Date                 Host        Tags        Directory
----------------------------------------------------------------------
562bfc5e  2018-07-14 20:18:01  mopped                  /home/user
bbacb625  2018-07-14 20:18:07  mopped                  /home/user
----------------------------------------------------------------------

Is there a way I can download just a “patch file” of the the diff between the two snapshots?

This is useful in cases where I’ve previously downloaded the first snapshot (which may be 100GB) and now want just the diff (which may be 2 MB).

Thank you.

It sounds like you have two copies of the repository: one local and one remote, somewhere in the cloud.
All in all I would say this: No you can’t just download the “patch file”.
I mean sure, you can download the blobs and the actual files a Restic repository is made out of, but that sounds like the wrong way to solve the problem you are seeing.

In the case the repository in the cloud is newer, as in it has a snapshot your local copy doesn’t, then you could “simply” your restic mount on the remote repository and get the file you are looking for there.
Even if you would save just the “diff” between the two snapshots, you won’t know for certain that the output you get represents the a whole file from start to finish - as in all data the file is made out of.

Maybe it’s more of interest to find out what you are trying to solve. Perhaps we can find a solution that works for you. :slight_smile:

If you want to apply the patch, your actual demand would be: “restore a snapshot, but use the existing data to prevent downloading things again”, right?

If this is what you want, you might want to have a look at

(The PR is not in a really good shape, I need to rebase it)

However, this PR does not remove files which are present in the target dir but not in the snapshot. This is the same behavior as current restore - maybe open an issue whether to add an option like --delete to the restore command`?

2 Likes

Thank you for the response! What I’m trying to solve is a case where I’ve downloaded the first snapshot (which is say, 100GB) and now want just the diff (which may be 2 MB), so that I don’t have to download the 100 GB again.

A patch file allows an existing file to be updated to reflect the latest state of the file, without downloading the whole file. Restic natively stores only diffs while backing up, and thus, in theory should be capable of this. Hope that explains :).

Ah ok, so what I was trying to get to with my post was something else. But the way you now responded makes it more clear to me what you meant. I was thinking too complicated or got confused as to how you “just” downloaded the first snapshot. But what you meant was that you have the actual data 1:1 which is in the first snapshot. :wink:

But as @alexweiss 100% correctly assessed the situation: this PR looks like exactly what you’re after :slight_smile:

1 Like

Ah, this is exactly what I was looking for, thank you!

Another way to think about the problem is this: it would be useful to have functionality that expresses the diff of two arbitrary snapshots as:

  1. data that is needed to transform one snapshot to another, plus
  2. operations (eg: delete, change timestamps, etc.) to transform one snapshot to another

With the above, one can do things like:

  • supply a local snapshot state, and have restic transform that into a requested snapshot state, while downloading the minimum of data required to do so (which is the main benefit). My original request above is a form of this
  • make restores resumable
  • restore only changed files

A slightly more generic way to think about this functionality is to add an restic rsync of sorts, which downloads the absolute minimum required to transform any arbitrary locally supplied state (not just an existing snapshot) to the requested restic snapshot.

Your PR addresses the main part of the functionality I’ve outlined above (and completely addresses my original request). --delete would be a part of (2) above.

For now, I’d love to see your PR merged. I’ve added the ideas above to github here.

Thank you again for the pointer and the PR, most appreciated!

I just realized there is already Option for restic restore to delete local files not in snapshot · Issue #2348 · restic/restic · GitHub to address --delete.