Trying to restore snapshots and getting Unknown blob

Getting Help
1

Hello everybody.
I need some help to restore data from a repo possibly corrupted.
I use restic v0.9.5. Repository is on an Openstack Swift cloud (OVH).

Backup script issue a forget command each days but never do a prune command. That make the repo huge, 1.5To. It contains 40 snapshots.
I’ve downloaded all datas on a separate server and began cleaning data with prune and rebuild-index.
Now the repo is lighter. A check command output:

check snapshots, trees and blobs
error for tree c9037c03:
  tree c9037c03: file "3693.dat.gz" blob 89 size could not be found
  tree c9037c03: file "3693.dat.gz" blob 90 size could not be found
  tree c9037c03: file "3693.dat.gz" blob 91 size could not be found
  tree c9037c03: file "3740.dat.gz" blob 31 size could not be found
  tree c9037c03: file "3740.dat.gz" blob 165 size could not be found
  tree c9037c03: file "3740.dat.gz" blob 178 size could not be found
  tree c9037c03: file "3740.dat.gz" blob 179 size could not be found
  tree c9037c03: file "3740.dat.gz" blob 271 size could not be found
  tree c9037c03: file "3740.dat.gz" blob 272 size could not be found
  tree c9037c03: file "3742.dat.gz" blob 22 size could not be found
  tree c9037c03: file "3750.dat.gz" blob 0 size could not be found
  tree c9037c03: file "3752.dat.gz" blob 99 size could not be found
  tree c9037c03: file "3752.dat.gz" blob 101 size could not be found
  tree c9037c03: file "3752.dat.gz" blob 115 size could not be found
  tree c9037c03: file "3752.dat.gz" blob 212 size could not be found
  tree c9037c03, blob c48928df: not found in index
  tree c9037c03, blob 73ee660c: not found in index
  tree c9037c03, blob 4e93af8a: not found in index
  tree c9037c03, blob 448c3d61: not found in index
  tree c9037c03, blob 534d2730: not found in index
  tree c9037c03, blob cb7d1d9f: not found in index
  tree c9037c03, blob 95d48cb8: not found in index
  tree c9037c03, blob 3deda202: not found in index
  tree c9037c03, blob 981f0747: not found in index
  tree c9037c03, blob c86ea602: not found in index
  tree c9037c03, blob 6b94347c: not found in index
  tree c9037c03, blob 97d9bec2: not found in index
  tree c9037c03, blob edba18d1: not found in index
  tree c9037c03, blob 3760e354: not found in index
  tree c9037c03, blob 413bf97d: not found in index
error for tree 9f0f7fca:
  tree 9f0f7fca: file "3698.dat.gz" blob 69 size could not be found
  tree 9f0f7fca: file "3700.dat.gz" blob 89 size could not be found
  tree 9f0f7fca: file "3700.dat.gz" blob 90 size could not be found
  tree 9f0f7fca: file "3700.dat.gz" blob 91 size could not be found
  tree 9f0f7fca: file "3700.dat.gz" blob 148 size could not be found
  tree 9f0f7fca: file "3700.dat.gz" blob 298 size could not be found
  tree 9f0f7fca: file "3700.dat.gz" blob 583 size could not be found
  tree 9f0f7fca: file "3700.dat.gz" blob 687 size could not be found
  tree 9f0f7fca: file "3700.dat.gz" blob 847 size could not be found
  tree 9f0f7fca: file "3700.dat.gz" blob 848 size could not be found
  tree 9f0f7fca: file "3700.dat.gz" blob 849 size could not be found
  tree 9f0f7fca: file "3769.dat.gz" blob 0 size could not be found
  tree 9f0f7fca, blob f986edd8: not found in index
  tree 9f0f7fca, blob c48928df: not found in index
  tree 9f0f7fca, blob 73ee660c: not found in index
  tree 9f0f7fca, blob 4e93af8a: not found in index
  tree 9f0f7fca, blob 30b83bc6: not found in index
  tree 9f0f7fca, blob 3199be1c: not found in index
  tree 9f0f7fca, blob d0871610: not found in index
  tree 9f0f7fca, blob bb33e0fb: not found in index
  tree 9f0f7fca, blob a07bc4f6: not found in index
  tree 9f0f7fca, blob 65913ec9: not found in index
  tree 9f0f7fca, blob 7b07cdbb: not found in index
  tree 9f0f7fca, blob 3aac0653: not found in index
  ...

Error happen on each snapshots. When i try to restore one i get the message:

Unknown blob c48928df2f19c6f7596324f8da318b67e02a0c6e298616c934d84c28a70141c6
github.com/restic/restic/internal/restorer.(*filePackTraverser).forEachFilePack.func1
        /restic/internal/restorer/filepacktraverser.go:21
github.com/restic/restic/internal/restorer.(*filePackTraverser).forEachFilePack
        /restic/internal/restorer/filepacktraverser.go:32
github.com/restic/restic/internal/restorer.newPackQueue
        /restic/internal/restorer/packqueue.go:52
github.com/restic/restic/internal/restorer.(*fileRestorer).restoreFiles

I searched the file data/c48928df2f19c6f7596324f8da318b67e02a0c6e298616c934d84c28a70141c6 in repo and it is missing.

I think 23 files/blob are missing from the repo.

Which possibilities i have to fully recover (or some part) the backup file ?
Can i do a repair action ?

Thanks for your help :pray: and I apologize for my english :slightly_smiling_face:

2

If you can, you should first try to find the root cause for the missing data file and eliminate it to make sure your backup is safe in future!

Moreover, your restic version is pretty old. I suggest you to update to 0.12.0. This version includes many bugfixes, is much faster and has improved checking and recovering abilities.

To heal your repo:

  • if you still have the file 3693.dat.gz, just run a backup which includes that file (with 0.12.0). As you already run rebuild-index, the index no longer contains the needed blobs and therefore they will be all stored again - healing your repo.
  • if you don’t have that file, it is unfortunately lost. You should still be able to restore the data from your repo if you exclude that file. In order to sanitize your repo, you have to either remove (forget) or repair all snapshots that contain that defect file. Repairing (i.e. modifying the snapshots such that the damaged file is removed/modified) is not contained in the released version, but you can try this PR and compile it yourself:
    Add repair command by aawsome · Pull Request #2876 · restic/restic · GitHub
3

Thanks Alexweiss for your help.

The root cause is from the destroyed datacenter of Strasbourg OVH. Their Openstack Storage was 3 duplicated but unfortunately 0.5% of the total storage had 3 copies inside the burnt building. I think theses blobs are in the 0,5%.

I have compiled and tried to issue a repair command. I was able to restore the data except 4 files.
I could rebuilt the missing data. That’s good!

I have another repository larger than this one and is impacted too. I haven’t done the rebuild-index yet. Should I do the rebuild-index before repair or directly repair it ?

1 Like
4

Sorry for the late reply…

If your repo is missing pack files, you should run a rebuild-index. With 0.12.0 this should be really fast!

Then I suggest you just run your standard backup(s) and see if this already heals your repo. If not, you can proceed with repair.

5

I have downloaded 0.12.0 to run the rebuild-index.
I was able to recover some parts of backup.
I rebuilt the lost parts from another backup. Now it is finished.
Thank you @alexweiss for your help :ok_hand: . You can set the topic as solved :wink: