Recovery from a repository with no snapshots

Getting Help
1

Hey, trying to recover data from a machine that is decommissioned. It had a single backup with restic in the hope to recover the data later.

The backup at the time was created with backrest and a test recovery was done at the time and succeeded.

The repository was found in a locked state, I took a full backup of the repository before unlocking it.

However it appears restic does not see any snapshots in the repository. The snapshots directory on the remote server is also empty.

$ restic snapshots
enter password for repository: 
repository 67511dae opened (version 2, compression level auto)
found 1 old cache directories in /home/[redacted]/.cache/restic, run `restic cache --cleanup` to remove them

The stats command states that no data is present:

$ restic stats --mode raw-data
enter password for repository: 
repository 67511dae opened (version 2, compression level auto)
found 1 old cache directories in /home/[redacted]/.cache/restic, run `restic cache --cleanup` to remove them
[0:06] 100.00%  192 / 192 index files loaded
scanning...
Stats in raw-data mode:
     Snapshots processed:  0
              Total Size:  0 B

However the list command does confirm there’s still some data in the repository to salvage:

$ restic list index
enter password for repository: 
repository 67511dae opened (version 2, compression level auto)
found 1 old cache directories in /home/[redacted]/.cache/restic, run `restic cache --cleanup` to remove them
5d4428181b9eb937893cc599e7c02ee03557ef8a2333a6ca16374ba66628c6f7
f8e3ed3f9afdcc680e55b104325ac6de649cd6eca09e88fcd2949db9eaa9cc59
a9c755cb8cdf4b2c920e783d5d932799fb39c6c00e5d33f8df318eab6c488a2a
a30e96958138c3312842c390bb60acf4bfabfb0c575b9648bdae33445b864bbe
8c586fec6ca57c77b5e840686c4f836dbde893bd776e864f7a7a459e5e6c1f99
[...]
$ restic list blobs
enter password for repository: 
data 60a3d4ad2a59488caf8ba30598478353f7f7d3bf1167d8d2c584e4d3024e3cf4
data 835b2877a6c60ff46b51ffb99da938ab5e78565b2746cc7fe229e3dd1033b077
data cfd65009998a5c5630a263ff25c3067536ceaa9a53a208854bbe2bfa5618a978
data 766c6343b077a623b7bcc9afeb59c873128568d75b55d98f5d6a3125502fa3d9
data c5f8477604840e8517df73bdbc439fcb9bad15ee91d8fc6f3850efdfa586bb07
[...]

Running a check with --read-data took an hour and revealed no errors:

$ restic check --read-data
using temporary cache in /tmp/restic-check-cache-1391535769
create exclusive lock for repository
enter password for repository: 
repository 67511dae opened (version 2, compression level auto)
created new cache in /tmp/restic-check-cache-1391535769
load indexes
[0:02] 100.00%  192 / 192 index files loaded
check all packs
15 additional files were found in the repo, which likely contain duplicate data.
This is non-critical, you can run `restic prune` to correct this.
check snapshots, trees and blobs
[0:00]          0 snapshots
read all data
[1:00:59] 100.00%  5534 / 5534 packs
no errors were found

Help in recovering this data would be much appreciated.

Using restic 0.17.3 compiled with go1.24rc1 on linux/amd64 from the fedora linux repositories.

2

Welcome! Taking a full backup of the repository was a great idea. I think that restic recover is the command you are looking for. Also, if this is an option in your system, I would upgrade to the latest version with restic self-update.

3

Done that earlier but forgot to mention that in the post, it didn’t actually do anything:

repository 67511dae opened (version 2, compression level auto)
found 1 old cache directories in /home/[redacted]/.cache/restic, run `restic cache --cleanup` to remove them
load index files
[0:00] 100.00%  192 / 192 index files loaded
load 0 trees
[0:00]          0 trees loaded
load snapshots
done

found 0 unreferenced roots
no snapshot to write.
4

Please try restic repair index --read-all-packs before restic recover.

5

This is now on the restic 0.18.0 binary from the github release.

Running restic repair index --read-all-packs resulted in:

getting pack files to read...
adding pack file to index 8550870523b384d9377daae6688e183f58e410f4141834012e5b199811c24ccf
adding pack file to index 85eba6a71a9fa9c561324432b44d8657148f0e8b40eabaa615be59d972478dec
adding pack file to index 8579891644d466c2e13109d6d963d17961800b5dedd70845d311f3d2556f8ba3
adding pack file to index 8596ffaca39dc9a92aa52df1df84ff976c9a7d751ea656d9644f135804901037
adding pack file to index 85deadb79710c255bc18685f5fdbd72152e4774478ba956128bfca3b193a4cb4
[...]
adding pack file to index d3a90ff25a8f683c60257f37cfec7d1bc5fd2b214f4f171b048ad78ffdfe42b9
adding pack file to index d39f6425fc9fa53afd6eab22e66bfd225af5f24bfc1f821ef77dc6ef90eaf38d
adding pack file to index d3e49258cb30feec6e93fa67eb5d842cf66584392d2c7a5e8a62155cb8930a99
adding pack file to index d3d25ba672dabded094b33e6ec5ac3a365f63f01e8b3cad5839985a20b04edc7
adding pack file to index d3e940bc630df661f80bdfbfe9d612c6ae467d2dcf932cd276bddb6920c01a43
reading pack files
[3:38] 100.00%  5549 / 5549 packs
rebuilding index
[0:00]          0 indexes processed
[0:00] 100.00%  192 / 192 old indexes deleted
done

The omitted output didn’t contain anything other than lines starting in ‘adding pack file to index’.

Running restic recover after didn’t do anything:

load index files
load 0 trees
[0:00]          0 trees loaded

load snapshots
done

found 0 unreferenced roots
no snapshot to write.
6

This is clearly beyond my knowledge at this point. I believe that someone more knowledgeable than me may help here.

I ran a simulation on a test repository by deleting the index and snapshots folders, and I was able to recover the files with the steps that I mentioned.

One thing that caught my attention is that restic check —read-data still returns 0 even without these directories on my test repository. Is it possible that this behavior is hiding a potential scenario where damaged repositories are not noticed?

The 192 / 192 old indexes deleted message gives me an impression that this repository can still be recovered somehow.

Since a test recovery was performed in the past, maybe not all files were copied from the repository? Can you still check if there are more files to be copied?

7

Since a test recovery was performed in the past, maybe not all files were copied from the repository? Can you still check if there are more files to be copied?

I don’t understand what you are asking, we no longer have access to the original system and believed to have a working repository as that same system could restore files back from the repository.

8

0 trees means that the repository does not contain any directory metadata at all. So all that’s left in the repository are the file data chunks, but there’s no information about which chunk belongs to which file.

I doubt that that backup run actually did complete. The only way I can imagine to create a repository with only data blobs but exactly zero tree blobs and no snapshots is by backing up a folder with some large files to an empty repository and then interrupting the backup before it has completed. (prune would also have removed all data blobs when there are no snapshots).

I suppose that neither the repository backup nor the repository contains a file in the snapshots folder? If both folders are empty, then the repository is in a different state than it was during the restore test.

9

I suppose that neither the repository backup nor the repository contains a file in the snapshots folder? If both folders are empty, then the repository is in a different state than it was during the restore test.

There is an empty snapshots folder on the locked copy, so suppose this was just user error at the time?

Is there anything we could do to partially recover the data?

10

According to your descriptions there previously was a snapshot file in the folder. However, I don’t see how it’s possible to get from that state to the one shown in the logs you’ve provided, unless the repository was completely wiped inbetween.

Well, you can run restic cat blob <blob hash> with the hashes output by restic list blobs, so for example restic cat blob 60a3d4ad2a59488caf8ba30598478353f7f7d3bf1167d8d2c584e4d3024e3cf4. That will extract the individual chunks from the repository. Anything larger than 1MB will likely be cut into multiple chunks so you’d have to somehow try to reassemble those manually. Other than that there’s nothing you can do as all the file metadata does simply not exist.

11

Better than nothing, suppose. Thank you for your help