Restore missing files. Best practices?

[juice]

How should I restore when I don’t know the specific snapshot that contains the files I want? Or actually, how should I restore a whole directory, when I’m not sure the latest snapshot contains that particular directory? I know a way, but I’d be interested to learn the right way.

Background

I made a restic backup /opt/foo/bar to test Restic and got snapshot 78e5ae96. Restic is great so I decided to backup more stuff, that is, everything in the parent directory /opt/foo. This is similar to a) in Understanding parent directories and subdirs and other understanding questions. Since then, I’ve been running the backup for the parent directory /opt/foo.

The /opt/foo/bar originally contained two files (a and b). Later, c was added, and currently there are three files:

/opt
  /foo
    /bar
      a.txt
      b.txt
      c.txt
    /dir2
      item.txt
    ...

The three files (a, b, c) were not modified before the latest snapshot, so the latest snapshot doesn’t contain them.

Problem

If I restic restore latest -i "foo/bar", the files are not restored as the latest snapshot doesn’t contain them.

I then did restic snapshots --path="/opt/foo/bar" and got the id of the original snapshot 78e5ae96. Ok, then:

restic restore 78e5ae96 -i "foo/bar"

I got files a and b, but no c. That makes sense because c wasn’t there originally. First I thought it’s a bug in the output because I’m not getting the latest snapshots with contents in the foo/bar directory but this excellent example clarified --path usage: Latest snapshot for criteria not found: no snapshot found

So, what is the best way to restore the whole directory /opt/foo/bar?

A Solution

restic find "foo/bar"

finds all snapshots that contain foo/bar. From the list I can grep the snapshots, and use the latest one:

restic restore 2816ba7c -i "foo/bar"

Now I get all three files (a, b, c).

Question

My process feels prone to errors as there are so many manual steps.

1. I need to understand and remember that restore latest snapshot doesn’t restore all data - only the data that’s referred to in the latest snapshot.
2. I need to know which data is missing and use restic find to find the relevant snapshots.
3. I need to parse restic find results to restore the latest snapshot that contains the desired data.

It works but I guess there’s probably a better way. Any thoughts?

[moritzdietz]

I don’t think restic should make the choice which version of a document (from which snapshot) it restores.
The default being the latest snapshot is a good default I feel like.
Also restic is made to be run interactively - so working with it and spending the time to compare states of a file from snapshot to snapshot to make sure which version is the correct one makes sense to me.

As far as your process goes it would be the way I would and did do it in the past.
It definitely requires you do combine a couple of restic commands to get the correct file back.

@juice

I need to know which data is missing and use restic find to find the relevant snapshots.

You can definitely run restic diff and compare snapshots

All in all this is a good summary.

[cdhowie]

@juice

The /opt/foo/bar originally contained two files (a and b). Later, c was added, and currently there are three files:

@juice

The three files (a, b, c) were not modified before the latest snapshot, so the latest snapshot doesn’t contain them.

This belies a misunderstanding of how restic works or your example is incomplete in some way – the snapshot should absolutely include these files if they were present at the time the backup was run, it will just re-use the blobs that contain the file data from prior snapshots. restic ls latest should verify this.

If it does not, then the files were excluded from the backup for some reason, but that reason is not that they weren’t modified since the last backup. It’s because at least one of these is true:

• They didn’t exist at the time of the most recent backup.
• The latest backup did not include the path where they reside.
• They matched at least one --exclude option and were therefore excluded.

Whether a file has changed since the most recent backup is not a factor in whether the file is included in a new backup.

[juice]

@cdhowie

the snapshot should absolutely include these files if they were present at the time the backup was run, it will just re-use the blobs that contain the file data from prior snapshots

Thanks! I was hoping someone would say that. This was my expectation and also the way I think it should work because then I would not need to keep track of (missing) files nor intervene manually when restoring.

Now the question becomes why I’m missing those files in restic restore.

1. restic ls latest does not list the files a, b, or c.
2. The three files did exist at the time of the most recent backup.
3. The latest backup should’ve included the path /opt/foo/bar. I’m running restic backup in the parent directory /opt/foo.
4. My excludes do not contain bar. However, a subfolder of bar is excluded.

Any other ideas? From the top of my head:

• Somehow Restic handles the /opt/foo/bar directory differently because it was initially backed up on its own. restic snapshots still lists the initial snapshot with the specific path even though there’s been hundreds of backups run for the parent directory since then.
• I’m on Mac, and have updated to MacOs v10.15.1 (Catalina). Catalina introduced new security features that prevent apps from accessing certain directories (ref, see “Security”). I don’t think restic is considered an “app”, but it would be great to hear from other Mac/Catalina users.
• It’s been a while since I’ve done restic prune. Should prune be run more often? Can it make a difference?

[cdhowie]

@juice

Somehow Restic handles the /opt/foo/bar directory differently because it was initially backed up on its own.

Prior snapshots are of no interest to restic during the backup process except to improve performance: if a “parent” snapshot exists, the metadata is used to avoid scanning the contents of files that are unlikely to have changed (if the inode, mtime, ctime, and size all match, for example). Note that directories are always scanned, so this would not explain any missing files.

If you want to force restic to re-scan the contents of all files, you can pass the --force flag to restic backup. This simply tells restic not to use a parent snapshot. In particular, --force should never result in different files being backed up, so I doubt it will solve your problem; its only use is to force the contents of all files to be re-hashed in cases where you know the metadata restic checks has not changed but the contents of some files might have. (It is very rare you should ever need to use this.)

@juice

It’s been a while since I’ve done restic prune . Should prune be run more often? Can it make a difference?

No, pruning will not affect which files restic stores in a snapshot.

All I can suggest is to upload a terminal session log where you run the following commands:

• find (to show the entire contents of the directory in question)
• restic backup -v 2 $YOUR_OPTIONS (and make sure to include the command itself in the output)
• restic ls latest after the backup has completed.

[moritzdietz]

@cdhowie

This belies a misunderstanding of how restic works or your example is incomplete in some way – the snapshot should absolutely include these files if they were present at the time the backup was run, it will just re-use the blobs that contain the file data from prior snapshots. restic ls latest should verify this.

I completely missed that part from the original post. Good catch!

[juice]

Thanks for the details. The usage of snapshots makes more sense now.

I ran the backup manually to make sure there are no strange errors. No errors, and 18 hours and a restic ls latest later, still the same result.

I’ll see if I can come up with a minimal example that would reproduce the functionality I’m experiencing.