How to restore previous version of file

Hi,

I am having trouble finding the previous version of a file in my restic repository.

Background
I have been using restic for backup for about 6 months. It seems to be working well, and I have spot-checked a couple of times to make sure that files are actually going into the repository and can be restored.

I create a snapshot every 6 hours, and then manage them using this cron job:

restic backup --exclude-file /etc/restic/restic-exclude /
restic forget --keep-hourly 13 --keep-daily 7 --keep-weekly 4 --keep-monthly 12 --keep-yearly 10 --prune

It all seems to work great. I have 24 snapshots in the repository going back to February.

The Problem
I unintentionally reinitialized our Wordpress website and went to restore it from backup.

But I cannot find any version of the site except for the most recent. This command:

restic -r /media/backup/stanley/restic-repo find /var/lib/docker/volumes/nextcloud_db_data/_data/mysql

reports that the files exist only in the most recent snapshot - the one made two hours ago, just after the unfortunate event:

Found matching entries in snapshot 686f5a37 from 2021-09-02 12:00:16

There appear to be no other versions of the files, even though I have backups going back to before they were first created.

How do I find the previous version? I only need to go back to whatever was there last week.

Thanks,
Brad

Thanks for such a detailed report!

Sounds odd - if that file wasn’t excluded by your exclude files/filters, it should have been backed up, at least assuming restic has been able to read the file - if it hadn’t it should have told you.

I suggest you use the snapshots command to identify a snapshot where you know the file should be in (i.e. before the unfortunate event we will not speak of), and then run the ls command with that snapshot ID. See also restic help ls for more info and options.

See if you can spot the file in the listing from that command, and perhaps also if you can glean some other info out of it (such as noticing that certain folders have been excluded).

Has there been any change or difference in how you’ve run the backups? I’m thinking of the remote possibility that you have had restic automatically running in a by which it hasn’t been able to read the file you need and you didn’t notice restic’s warnings about it.

Thanks for the super-quick response, rawtaz!

Answer to your question
No change to how I run the backups that I recall. I set it up months ago and it’s just been humming along. But I suppose I can’t say 100% for sure.

It’s not in any other snapshots
I believe I have tried what you suggested. I looked in a snapshot from a week ago, and the file isn’t there. I have also tried the thing of doing a restic mount and then (within the mounted path) doing ls [mountpoint]/snapshots/*/path_to_file. The only instance that is found is the one from earlier today.

I began to fear the following scenario, but it seems to me that it wouldn’t make any sense for it to work this way:

  • The previous version of the file (actually a whole folder) was backed up in a snapshot made just after it was created.
  • The file has never changed since.
  • The snapshot was forgotten and pruned according to the forget-and-prune schedule
  • The file only appeared in the snapshot in which it was first created, so now it’s gone forever.

But that scenario doesn’t make sense to me - if it worked that way, nobody would ever want the forget/prune feature.

More detail
Here’s a look into the folder in question, inside the “latest” snapshot in the mount:

drwx–x–x 2 root root 0 Sep 2 09:41 ./
drwxr-xr-x 2 root root 0 Mar 29 16:44 …/
drwx–x–x 2 root root 0 Mar 27 22:28 buildkit/
drwx-----x 2 root root 0 Sep 2 09:48 containers/
drwx------ 2 root root 0 Mar 27 22:28 image/
drwxr-x— 2 root root 0 Mar 27 22:28 network/
drwx-----x 2 root root 0 Sep 2 09:48 overlay2/
drwx------ 2 root root 0 Mar 27 22:28 plugins/
drwx------ 2 root root 0 Sep 2 09:41 runtimes/
drwx------ 2 root root 0 Mar 27 22:28 swarm/
drwx------ 2 root root 0 Sep 2 09:48 tmp/
drwx------ 2 root root 0 Mar 27 22:28 trust/
drwx-----x 2 root root 0 Sep 2 09:48 volumes/

You can see that this stuff was all originally created on March 27, while the overwrites occurred today. And the thing I need is “nextcloud_db_data” within volumes:

root@stanley:/mnt/repo/snapshots/latest/var/lib/docker/volumes# ll
total 33
drwx-----x 2 root root 0 Sep 2 09:48 ./
drwx–x–x 2 root root 0 Sep 2 09:41 …/
brw------- 1 root root 0, 0 Sep 2 09:41 backingFsBlockDev
drwx-----x 2 root root 0 Sep 2 09:48 c59f2d58c3a0d647c03d71844135eb0c977c30d0023f239de8b4563adb8879ed/
drwx-----x 2 root root 0 Sep 2 09:48 ed40321be6b07fffccedea413d4ada604f222d09bf6b1c0f20ea9028566bf7e4/
drwx-----x 2 root root 0 Sep 2 09:48 f20bb0a6a0eb6583073ee432cd1c1340a7470f2cba21badd18951adf10c20f43/
-rw------- 1 root root 32768 Sep 2 09:48 metadata.db
drwx-----x 2 root root 0 Sep 2 09:46 nextcloud_db_data/

And you can see here that I have a snapshot from March 31, so you’d think it would be in there:

root@stanley:/mnt/repo/snapshots# ls
2021-02-28T18:00:15-08:00 2021-08-28T18:00:15-07:00 2021-09-01T06:00:15-07:00
2021-03-31T18:00:15-07:00 2021-08-29T18:00:15-07:00 2021-09-01T12:00:15-07:00
2021-04-30T18:00:16-07:00 2021-08-30T12:00:15-07:00 2021-09-01T18:00:15-07:00
2021-05-31T18:00:15-07:00 2021-08-30T18:00:15-07:00 2021-09-02T00:00:15-07:00
2021-06-30T18:00:15-07:00 2021-08-31T00:00:15-07:00 2021-09-02T06:00:15-07:00
2021-07-31T18:00:15-07:00 2021-08-31T06:00:15-07:00 2021-09-02T12:00:16-07:00
2021-08-15T18:00:16-07:00 2021-08-31T12:00:16-07:00 latest
2021-08-22T18:00:15-07:00 2021-08-31T18:00:15-07:00
2021-08-27T18:00:15-07:00 2021-09-01T00:00:15-07:00

But it simply isn’t there. The “volumes” folder is there, but inside it there is no folder “nextcloud_db_data”.

root@stanley:/mnt/repo/snapshots/2021-03-31T18:00:15-07:00/var/lib/docker/volumes# ll
total 33
drwx-----x 2 root root 0 Mar 29 23:34 ./
drwx–x–x 2 root root 0 Mar 29 23:34 …/
brw------- 1 root root 0, 0 Mar 29 23:34 backingFsBlockDev
-rw------- 1 root root 32768 Mar 29 23:34 metadata.db

So then my aforementioned fear sets in. What if it was in a snapshot from 3/27/2021 which then got blown away?

When snapshots are forgotten and pruned, is the process sophisticated enough to move unique content from the pruned snapshots forward to a later snapshot?

If it isn’t that sophisticated, am I using the tool incorrectly, and/or trying to ask it to do something it isn’t intended to do?

If you say no, my understanding is correct and it should work the way I think, then I’ll just chalk it up to my having shot myself in the foot at some point doing something that I don’t recall. I suppose that in the early days I could have done something naive like manually deleting a snapshot from the repo rather than using restic to do it, although I don’t know why I would have done that.

Of course, if you think my data might still be in there somewhere, that would be best of all, but I’m beginning to lose hope for that.

Thanks,
Brad

I’m on the run so I can’t answer all of your last reply, but I agree that it’s looking rather dark since you’re not seeing the file and since you have run the prune command which would wipe out unreferenced data. You can always try the recover command just to see if it gives you anything.

In general, restic saves a reference for each file it finds and that isn’t excluded, so assuming it sees that file it should save it and keep it in every snapshot (deduplicated) for as long as the file exists. It’s only when the file is no longer seen by restic (e.g. because it’s deleted in the filesystem) that it is no longer included in the snapshot for a backup run. Until then, every snapshot will reference the same blocks of data that were backed up for that file in the very beginning, and even if you remove an older snapshot where the file was referenced, the more recent snapshots that still reference the file will keep their reference and the data for the file will not be removed.

So at this point I can only presume that either you’ve done something wrong or (more likely) there’s something about the filesystem that has made restic not see that file or its folders. Doesn’t Docker do some funky stuff with its volumes? I know there are good reasons why people say only work with volumes through the Docker CLI and not directly in the filesystem, perhaps you got bit by that.

Can you please pastebin the complete environment variables and the exclude files involved, and if you haven’t already, the complete commands that have been used to backup, forget and prune? Just so we can see the complete picture.

Also, which version(s) of restic have you been/are you using?

And finally, perhaps you already told us, but have you at any point verified that the nextcloud_db_data folder and the files you’re looking for have actually been present in any snapshot, before the last one after the unfortunate event we will not speak of?

I’ll follow up further when I’m sure, but it now looks like this is not a restic problem at all.

Rather, it appears related to

  • Some fundamental misunderstanding on my part of how docker works
  • A change in behavior between versions of something in my docker system

Mystery Solved
Specifically I finally found the “missing” files in my ${NEXTCLOUD_ROOT} rather than under docker’s “volumes”.

Somehow, previously, my mysql database was being stored under ${NEXTCLOUD_ROOT}, whereas after the updates to all my containers, it’s now being stored in “volumes”. So the missing files had never been where I was looking for them.

Indeed, I do find the original March 29 files in restic, when I look for them under ${NEXTCLOUD_ROOT}.

Thanks for Your Help
I’m 95% sure this is the problem and that restic is completely exonerated. I apologize for the confusion, but thanks for your help. Your input was actually helpful in chasing this down, even though it appears that it didn’t have anything to do with restic itself.

Thanks,
Brad

The final truth was somewhere in between. I wasn’t quite as daft as I thought.

The folder in ${NEXTCLOUD_ROOT} was not active - that was leftover from an early experiment.

But it did inspire me to search. What I learned is that the location of docker volumes has changed. Until today, they were under

/var/snap/docker/common/var-lib-docker/volumes

but they are now under

/var/lib/docker/volumes/

And not only that, I was able to find the previous website in restic, HOORAY!

Thanks for such a great tool.

Brad

2 Likes

:+1: :champagne: