A common use case of backup software for me is to restore an individual file that I found has been messed up. This is straightforward to do with restic, given the problem becomes apparent before any further snapshots are taken - just restore the latest version. Yet, when the messed-up file has been sitting there for some time, with multiple snapshots taken inbetween, it becomes cumbersome to identify the exact snapshot that contains the file in its previous, healthy state.
So what I would like to see is a history of a particular file across snapshots, that clearly identifies when in the past that file has changed. This would allow me to select a snapshot from which I can then restore the file. I think, currently this is possible only by manually browsing the mounted repository, checking the file in each snapshot, going back in time one by one.
This feature can be thought of as an extension of the find command, adding the following information/processing:
in addition to the “snapshot contains file” information the output should include the file’s hash in each containing snapshot
the snapshot’s timestamp should be given
the resulting list of hash-per-snapshot records should be sorted by timestamp and grouped by hash, so that only the youngest snapshot containing the file with a particular hash is shown
Does this make sense to others? Would that be hard to implement?
For a backup solution restoring files is of course equally important as saving them, hence the restore process should be easy and comfortable. While restic could be improved in this regard by providing more information it should also be as simple as possible.
Is this information really necessary? Hardly anybody knows the hash of the file they are searching for. Most people will use restic find --oldest DATE --newest DATE.
While I agree that this information would be nice to have I don’t think it is necessary. When using restic find --long the modification time of the file is displayed, which usually is all you are looking for.
You are right. Currently there doesn’t seem to be any kind of sorting which is kond of confusing.
Thanks, @764287, for pointing out the --long option to the file command; I was not aware of it, and it helps.
Yet, instead of the current output, which seems to be organized by snapshot first, then by file match
$ restic find --long /home/marcus/.xsession-errors
repository 2a7fabeb opened successfully, password is correct
Found matching entries in snapshot d95b4a13
-rw------- 1000 1000 581436 2018-08-26 17:49:27 /home/marcus/.xsession-errors
Found matching entries in snapshot ded778cf
-rw------- 1000 1000 557607 2018-08-26 12:55:07 /home/marcus/.xsession-errors
Found matching entries in snapshot f6814a59
-rw------- 1000 1000 545425 2018-08-27 20:27:41 /home/marcus/.xsession-errors
I’d find the following, grouped by file match first, then by snapshot, much more useful:
repository 2a7fabeb opened successfully, password is correct
Found matching entries for file /home/marcus/.xsession-errors
snapshot f6814a59: 2018-08-27 20:27:41 545425 -rw------- 1000 1000
snapshot d95b4a13: 2018-08-26 17:49:27 581436 -rw------- 1000 1000
snapshot ded778cf: 2018-08-26 12:55:07 557607 -rw------- 1000 1000
If sorted, this represents a useful file history. Then, to make it more concise, an extra option (like --unique) could drop records that apparently represent the same file state, by modtime. I am with @cfbao in that I’d rather not rely on modtime and prefer the content hash instead, but as restic IIRC does select files for backup by modtime before computing their hashes, it would be pointless to be more strict for the file command.
@BenBipodfind takes a pattern so the file path may be different for each match, and they may be multiple matching files in each snapshot. So your example is an edge case for find output.
But there could be a compact one-line output for find. Did you test find with the --json option? If it produces JSON output you could format it on to one line with jq
@whereisaaron, I’m aware of my example being an edge case as, typically, I’d want the history of a specific file and not of a file pattern.
Thanks for pointing out the JSON-Option and jq, that might be the way to go for a work-around.
Well, my idea was that pattern matching still applies, but that snapshot and file path change places. Currently, the output has pattern matches per snapshot and i would like this to be snapshots per pattern match (not really caring for the pattern match myself, as in my use case I would use it with an absolute path). Having a more compact output than the current one is a step in the right direction, but doesn’t do the trick; sorting is essential.
And you’re right, my example would certainly be more convincing with .ssh/id_rsa
There may be a pragmatic reason for snapshot order rather than file match order. My guess is find is looping through the snapshots is date/time order and outputting matches as soon as it finds them.
The find doesn’t know what files matches it will find until it has looped through all snapshots. So to achieve your proposed file-path-sorted output it would have to delay output and remember all matches in all snapshots, then sort and output those results after it has finished. A very broad (e.g. *.jpg) pattern might require caching a lot of data to sort and display at the end of the process.
Your reasoning seems totally applicable to me. While I tried to be unobtrusive by asking for a change to an existing command, the history use case and the find command don’t appear to be the ideal match. So have a new history command, similar to find but for unique, full-path files only?
I vaguely remember that the inclusion of the snapshot date in restic find --long was discussed (and agreed upon?) in a GitHub issue.
Do I remember correctly? If someone knows what I’m talking about, could you post a link?
If snapshot date is included, I think file hash is the only thing missing for restic find to work as a simple file version history tracker.
I just ran a restic find --no-lock "/home/myusername/sites/backups/mysqldb/wp*_5000.sql.gz" command. Files were found in many different snapshots, but the snapshots are listed in no discernable order. The first snapshot listed was from 2021-05-18. The next one was from 2021-04-30. The final one (of many others) was from 2021-05-23. And somewhere in between, there was one from today (2021-05-24)!
The snapshots are not listed in the order in which they were created, nor in the order of their hashes. Seems to be no logic to the sorting.
In addition, the find command took quite a long time – several minutes. I’m not sure what performance to expect, but I was thinking it would be speedier, like an indexed database search. (restic stats: Snapshots processed: 33. Total File Count: 3222114. Total Size: 14.050 GiB.)
I would much prefer it if restic find listed matches in descending order of the snapshot date. That has the advantage that (a) the newest matches are listed first, so I can quit if I don’t want to wait for more, and (b) it effectively gives me a version history if I want to look for versions of a specific file.
