Restic migrate reports error in tree that no snapshot contains

Getting Help
1

I am trying to modernize an existing restic repository using restic 0.14.0. Initial checks report a missing data file, and then an error in one specific tree:

% restic migrate upgrade_repo_v2

repository 6f6b6180 opened (repository version 1) successfully, password is correct
checking repository integrity...
using temporary cache in /tmp/restic-check-cache-2941017919
repository 6f6b6180 opened (repository version 1) successfully, password is correct
created new cache in /tmp/restic-check-cache-2941017919
load indexes
check all packs
pack a6e17d0139b44785e403f2ca8b5bcf56067d6ec6f073c2e3a44dfeb19775a566: not referenced in any index
pack 58e7e5f3d3e43c3e5fde12cde1d7a9b0a24311f09f7a01c2aac4762226e39811: not referenced in any index
pack 4b78c6e99fbcbf7e1389f3e6eb226e6ffe30ae25e1196ef2dd4b24abb70bb927: not referenced in any index
3 additional files were found in the repo, which likely contain duplicate data.
This is non-critical, you can run `restic prune` to correct this.
check snapshots, trees and blobs
Load(<data/81cb27601a>, 0, 0) returned error, retrying after 582.280027ms: <data/81cb27601a> does not exist
Load(<data/81cb27601a>, 0, 0) returned error, retrying after 703.28564ms: <data/81cb27601a> does not exist
Load(<data/81cb27601a>, 0, 0) returned error, retrying after 1.040217184s: <data/81cb27601a> does not exist
Load(<data/81cb27601a>, 0, 0) returned error, retrying after 2.002763936s: <data/81cb27601a> does not exist
Load(<data/81cb27601a>, 0, 0) returned error, retrying after 1.431768704s: <data/81cb27601a> does not exist
Load(<data/81cb27601a>, 0, 0) returned error, retrying after 2.492721545s: <data/81cb27601a> does not exist
Load(<data/81cb27601a>, 0, 0) returned error, retrying after 3.399927963s: <data/81cb27601a> does not exist
Load(<data/81cb27601a>, 0, 0) returned error, retrying after 6.842164996s: <data/81cb27601a> does not exist
Load(<data/81cb27601a>, 0, 0) returned error, retrying after 13.00939464s: <data/81cb27601a> does not exist
Load(<data/81cb27601a>, 0, 0) returned error, retrying after 25.250366555s: <data/81cb27601a> does not exist
Load(<data/81cb27601a>, 927, 711649) returned error, retrying after 357.131936ms: <data/81cb27601a> does not exist
Load(<data/81cb27601a>, 927, 711649) returned error, retrying after 660.492892ms: <data/81cb27601a> does not exist
Load(<data/81cb27601a>, 927, 711649) returned error, retrying after 920.315446ms: <data/81cb27601a> does not exist
Load(<data/81cb27601a>, 927, 711649) returned error, retrying after 1.635001613s: <data/81cb27601a> does not exist
Load(<data/81cb27601a>, 927, 711649) returned error, retrying after 1.982055195s: <data/81cb27601a> does not exist
Load(<data/81cb27601a>, 927, 711649) returned error, retrying after 3.011308614s: <data/81cb27601a> does not exist
Load(<data/81cb27601a>, 927, 711649) returned error, retrying after 6.715255694s: <data/81cb27601a> does not exist
Load(<data/81cb27601a>, 927, 711649) returned error, retrying after 6.138576273s: <data/81cb27601a> does not exist
Load(<data/81cb27601a>, 927, 711649) returned error, retrying after 9.010955269s: <data/81cb27601a> does not exist
Load(<data/81cb27601a>, 927, 711649) returned error, retrying after 16.547394626s: <data/81cb27601a> does not exist
error for tree cb5e822d:snapshots
  ReadFull(<data/81cb27601a>): <data/81cb27601a> does not exist
[19:51] 100.00%  78 / 78 snapshots

My usual scorched-earth recovery strategy would be to restic forget all snapshots that contain the defective tree. However, no snapshot seems to contain this tree:

% restic find --tree cb5e822d

repository 6f6b6180 opened (repository version 1) successfully, password is correct

# Executes for 8 minutes, but produces no other output.

How can I get my repository back to a valid state? Forgetting entire snapshots is fine with me, if that’s the easiest way to get there. But I’d rather not throw away the entire repository, if something less extreme will work.

2

Have you tried restic find --blob 81cb27601a?

3

I just tried “restic find --blob 81cb27601a” per @rawtaz’s suggestion. It ran for 8 minutes and produced no output beyond the initial “repository opened” message.

4

It’s been too long since I had integrity problems in my repositories. Sorry. Stand by for someone more clueful to help!

5

Thanks for taking a shot at it, @rawtaz!

6

Seems like you are having a problem with your index here - you have files not referenced by the index (this can have multiple reasons and some are non-critical, but one is a wrong index) AND when traversing the tree, restic tries to load data from pack files which obviously don’t exist (which does indicate that the index doesn’t reflect the pack file situation).

So I advise you to run rebuild-index first (using a recent restic version). There are most likely more errors due to missing pack files but you can only analyze them when the index is corrected.

7

Thanks for this suggestion, @alexweiss. Unfortunately, it didn’t help.

restic rebuild-index completed without error. restic migrate upgrade_repo_v2 fails as before, reporting missing data/81cb27601a and an error for tree cb5e822d:

% restic migrate upgrade_repo_v2

repository 6f6b6180 opened (repository version 1) successfully, password is correct
checking repository integrity...
using temporary cache in /tmp/restic-check-cache-20130587
repository 6f6b6180 opened (repository version 1) successfully, password is correct
created new cache in /tmp/restic-check-cache-20130587
load indexes
check all packs
check snapshots, trees and blobs
Load(<data/81cb27601a>, 0, 0) returned error, retrying after 582.280027ms: <data/81cb27601a> does not exist
Load(<data/81cb27601a>, 0, 0) returned error, retrying after 703.28564ms: <data/81cb27601a> does not exist
Load(<data/81cb27601a>, 0, 0) returned error, retrying after 1.040217184s: <data/81cb27601a> does not exist
Load(<data/81cb27601a>, 0, 0) returned error, retrying after 2.002763936s: <data/81cb27601a> does not exist
Load(<data/81cb27601a>, 0, 0) returned error, retrying after 1.431768704s: <data/81cb27601a> does not exist
Load(<data/81cb27601a>, 0, 0) returned error, retrying after 2.492721545s: <data/81cb27601a> does not exist
Load(<data/81cb27601a>, 0, 0) returned error, retrying after 3.399927963s: <data/81cb27601a> does not exist
Load(<data/81cb27601a>, 0, 0) returned error, retrying after 6.842164996s: <data/81cb27601a> does not exist
Load(<data/81cb27601a>, 0, 0) returned error, retrying after 13.00939464s: <data/81cb27601a> does not exist
Load(<data/81cb27601a>, 0, 0) returned error, retrying after 25.250366555s: <data/81cb27601a> does not exist
Load(<data/81cb27601a>, 927, 711649) returned error, retrying after 357.131936ms: <data/81cb27601a> does not exist
Load(<data/81cb27601a>, 927, 711649) returned error, retrying after 660.492892ms: <data/81cb27601a> does not exist
Load(<data/81cb27601a>, 927, 711649) returned error, retrying after 920.315446ms: <data/81cb27601a> does not exist
Load(<data/81cb27601a>, 927, 711649) returned error, retrying after 1.635001613s: <data/81cb27601a> does not exist
Load(<data/81cb27601a>, 927, 711649) returned error, retrying after 1.982055195s: <data/81cb27601a> does not exist
Load(<data/81cb27601a>, 927, 711649) returned error, retrying after 3.011308614s: <data/81cb27601a> does not exist
Load(<data/81cb27601a>, 927, 711649) returned error, retrying after 6.715255694s: <data/81cb27601a> does not exist
Load(<data/81cb27601a>, 927, 711649) returned error, retrying after 6.138576273s: <data/81cb27601a> does not exist
Load(<data/81cb27601a>, 927, 711649) returned error, retrying after 9.010955269s: <data/81cb27601a> does not exist
Load(<data/81cb27601a>, 927, 711649) returned error, retrying after 16.547394626s: <data/81cb27601a> does not exist
error for tree cb5e822d:snapshots
  ReadFull(<data/81cb27601a>): <data/81cb27601a> does not exist
[17:29] 100.00%  79 / 79 snapshots
Fatal: repository contains errors

Also as before, restic find runs for eight minutes but reports nothing for the missing blob:

% restic find --blob 81cb27601a

repository 6f6b6180 opened (repository version 1) successfully, password is correct

Same goes for the erroneous tree:

% restic find --tree cb5e822d

repository 6f6b6180 opened (repository version 1) successfully, password is correct
8

Which backend are you using? Can you check if a file in data/81/ exists whose name starts wit h 81cb27601a? As running rebuild-index did not remove the reference to that pack file, that looks like the backend claims that the file exists, but accessing it is somehow not possible.

restic find --pack 81cb2760 should show which snapshots depend on that pack file.

9

Thanks for weighing in, @MichaelEischer.

Which backend are you using?

I’m using the rclone backend, which in turn is using its pcloud backend.

Can you check if a file in data/81/ exists whose name starts with 81cb27601a ?

Yes: data/81/81cb27601a78e7e5a5782f65b40f66d2ad0ba781b18c16dfd7bae1fa1855c63a exists. I can read its contents via the standard pCloud desktop client. However, trying to read its contents via the standard pCloud web interface yields a “404 Not Found” error! So this seems to be some sort of data-integrity problem with pCloud. That’s troubling, though presumably not Restic’s fault.

restic find --pack 81cb2760 should show which snapshots depend on that pack file.

That command ran for ten minutes but produced no output beyond the initial “repository opened” message.

Given that this data/81/81cb27601a78e7e5a5782f65b40f66d2ad0ba781b18c16dfd7bae1fa1855c63a file is readable via the pCloud desktop client but not the pCloud web interface, I decided to use the desktop client to copy it out of pCloud entirely, then back in. Perhaps that would heal whatever damage was affecting pCloud’s web interface? Nope. This file is now absent from the repository no matter which pCloud client software I use. Oh well. At least now, one hopes, it is consistently gone.

I used restic rebuild-index again in case that would help Restic adapt to this file’s disappearance. Index rebuild completed without error. However, restic migrate upgrade_repo_v2 still fails with the same diagnostics as before:

Load(<data/81cb27601a>, 0, 0) returned error, retrying after 582.280027ms: <data/81cb27601a> does not exist
...
error for tree cb5e822d:snapshots
  ReadFull(<data/81cb27601a>): <data/81cb27601a> does not exist

restic find --pack 81cb2760a reports an error not seen earlier:

Fatal: unable to find pack(s): [81cb2760a]

restic find --blob 81cb2760a reports nothing beyond the initial repository opened message.

restic find --tree cb5e822d reports nothing beyond the initial repository opened message.

Any suggested next steps to repair this repository?

10

Manually copy your whole repository to a trustworthy storage before you continue…
This could be your local hard drive or some other remote storage.

11

That sounds like it would be a good idea to open a support ticket at pCloud. It’s not the first time that we see problems with pCloud here (although these were a bit different).

So I guess you still have a copy of that file? That would definitely help with repairing the repository.

The output of restic shows that the file is nevertheless returned when restic lists all files in the surrounding folder.

To move forward we have to get rid of this broken file. What might be worth a try is to copy / move all files from data/81 to a new folder. Then delete the old folder. And afterwards move the files back. Depending on how pCloud represents folders, that might help or not. Ideally you first make a full copy of that folder just to be safe.

If the filename is permanently broken, then I’m afraid you’ll have to rename the repository, which should avoid the broken filename. Or maybe the support is able to fix things.

The id returned by ReadFull(<data/ ID >) is a pack ID so the other commands cannot work. find --pack currently only accepts short pack IDs with 8 characters or full ones with 64 characters. That is you’d have to run restic find --pack 81cb2760 (as mentioned in my last reply) or restic find --pack 81cb27601a78e7e5a5782f65b40f66d2ad0ba781b18c16dfd7bae1fa1855c63a`.

12

Wise advice. Copying is in progress, but has run into multiple errors suggesting problems on pCloud’s side of things. I’m still working on this, and won’t make any further changes to the original repository until I have a safe, trustworthy copy elsewhere. Thanks, @alexweiss.

13

Indeed! I have a ticket open with them now. They “fixed several diff problems related to the account”. However, my initial attempt to copy down the entire repository failed for ~45 files, and those files are still failing now. So pCloud has some more recovery work to do.

I am suspending any Restic-level efforts to repair this repository until pCloud has either recovered my files or definitively told me they cannot.

More news here when I have it. Thanks for your efforts so far, @MichaelEischer, @alexweiss, and @rawtaz!

1 Like