Files not found in index

Hello, I use restic for about 6 months to make backups to an external HD, every backup followed by a check to ensure it’s ok and it has been amazing.

Current version: restic 0.12.1 compiled with go1.16.6 on windows/amd64

Yesterday it complained about packs that does not exist:

using temporary cache in C:\cygwin64\tmp\restic-check-cache-023509963
enter password for repository:
repository 82505644 opened successfully, password is correct
created new cache in C:\cygwin64\tmp\restic-check-cache-023509963
create exclusive lock for repository
load indexes
check all packs
pack b2fa5462: does not exist
pack dfaa5f6e: does not exist
pack b2ff416e: does not exist
pack c109f44f: does not exist
pack b2fbd671: does not exist
pack dfac5f73: does not exist
pack b2febf5e: does not exist
pack b2fc8283: does not exist
pack dfac54da: does not exist
pack ed3b77ad: does not exist
pack dfade653: does not exist
pack dfad2b4c: does not exist
pack dfaa9fbb: does not exist
pack 7bd887c8: does not exist
pack b2fedd29: does not exist
pack b2fed0dd: does not exist
pack 0566d628: does not exist
pack dfaa5f66: does not exist

So i’ve made a backup of the index folder and ran a rebuild-index, it removed those pack files and now i can see wich files were affected:

 restic -r . check
using temporary cache in C:\cygwin64\tmp\restic-check-cache-231728407
enter password for repository:
repository 82505644 opened successfully, password is correct
created new cache in C:\cygwin64\tmp\restic-check-cache-231728407
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
error for tree c9134970:napshots
  tree c9134970: file "edb0000B.log" blob f85c45d74df17375d27557390f69d13cf8c625bf74e4fb67f0697e9f07ca1cab not found in index
  tree c9134970: file "edb0000B.log" blob 9607b82ffc51bf6bedae79cfada33f6d56ae730715189f1f3a3f5198f1a85fa0 not found in index
  tree c9134970: file "edb0000C.log" blob 162560bc4845a28a9fafae369d70008456f30908a57160bf6fd052d25c5bc1de not found in index
  tree c9134970: file "edb0000C.log" blob 7742dda9475b52f677ebf12cc1a66c9f092ad937bdf8254a673c39ea489d57eb not found in index
  tree c9134970: file "edb0000C.log" blob dfb2467edd19f8f77dd9bb53d3e2af1b03332e4cc65ade6e40fbfdcd17bd2be2 not found in index
error for tree e3a50137:napshots
  tree e3a50137: file "client.lzma" blob 627562212102da27d1e6b8fbb4126180ad3fd84eb53f89db4ca9dbf4cf9885bb not found in index
error for tree f9272c4e:
  tree f9272c4e: file "client.lzma" blob 627562212102da27d1e6b8fbb4126180ad3fd84eb53f89db4ca9dbf4cf9885bb not found in index
error for tree ab166c91:napshots
  tree ab166c91: file "pendrive.7z.003" blob 6c057a068ab4f8fd267fa44bce9f489ec87768725203b04b055dbe5b6599deca
not found in index
  tree ab166c91: file "pendrive.7z.003" blob 72f005d35eb5679d6e64908a3be745555bb2afd8da1dbd78135a56459f051979
not found in index
error for tree 06b9cdc6:napshots
  tree 06b9cdc6: file "VID_20161027_093710156.mp4" blob 3a6d0108deef6f432b2b11af98c0ab7c046231be49f6c44aa99cb329a42d4906 not found in index
  tree 06b9cdc6: file "VID_20161027_094636330.mp4" blob 8b92a8f63b705dbeea3ebf73007caf685d4c0144a996fc73239f8ff9be8f2cf8 not found in index
error for tree a3362f61:napshots
  tree a3362f61: file "Sculpting Course.zip" blob 8d382b1f54ae16e1b494af42a132e204caf6de383337489efb5f6e73f84f9041 not found in index
  tree a3362f61: file "Sculpting Course.zip" blob 962fb49c77cc777ee171ffd5e78f4b7dca8bb5012eaf47533b1c325f4d759266 not found in index
error for tree c6ed7843:napshots
  tree c6ed7843: file "Backup Arext.7z" blob ea4bd174942ac68e49a05049c35ddbade3dfc2dc326cc6fa2a9877262d82a872 not found in index
  tree c6ed7843: file "Backup Arext.7z" blob cfefbcdfa9ca23e34b60392c50c5e7b3ceb4e8b8fa15bc7beb3acda290745032 not found in index
  tree c6ed7843: file "Backup Arext.7z" blob 3c0864dc9f0629326f527c76572d4af17750c03702032fc5b586d40cdbde66fc not found in index
  tree c6ed7843: file "Backup Arext.7z" blob 60ff77497fc413d104635c5fad91a681fa8594cb93d2ef3a9d8ba20d5cc6b29c not found in index

  tree c6ed7843: file "Backup Arext.7z" blob a625ae26cf15b5fc84a56dcde6bbdb319990c55b6ad7d926bc00e5e3f10dc101 not found in index
  tree c6ed7843: file "Backup Arext.7z" blob 388d6ecf0df4393892ec68a8f8553cc5eae7f57309aa5157ef9eae6b658f8acb not found in index
  tree c6ed7843: file "Backup Arext.7z" blob f1ba9ffc52ed18116db5c14bf97cbf0a3379e9005a70aebacd618665e85c5d33 not found in index
  tree c6ed7843: file "Backup Arext.7z" blob 411c768f8e6d11d4318d306f6ff017d2c1a294b9692e826f4d19a7d56b9bd81b not found in index
Fatal: repository contains errors

I repaired a few files with a new backup of the system. (the ones on the list are the few that I do not have outside of the repo)

I have no idea about how these files got in this state, the s.m.a.r.t of the disk says it’s ok and I have a reasonable guess that the files are still inside the repository because each .zip and .7z file are about ~30-50gb and the overall disk space did not change.

I have yet to run a check with --read-data because it will probably take 2 days to complete but the 5% random subset led me to believe the rest is mostly ok, but it’s on my to-do list to verify the integrity of all the data.

The damage seems to be random, the zip and .7zip files are old, one of the first snapshots i’ve ever made, while the logs and .lzma’s are a few cache files I made this week (and would be pruned later this week).

So I don’t know if I can trust my external HD anymore and don’t know if these files are recoverable, I’d appreciate some help.

Which restic commands did you run between the last successful check run and the failed check run? Only backup or also prune, unlock or something else? Did you notice any errors reported in prior restic runs or did something unexpected happen? Maybe the HDD was not cleanly removed or did the host go into standby while restic was running?

What did rebuild-index report? I’d assume that is just reported that it has removed the missing pack files from the index? (restic find --pack b2fa5462 is since restic 0.12.1 also able to report which snapshots are affected by the missing pack files, without having to run rebuild-index first).

What strikes me as odd are that most missing pack files are from the same two data folders in the repository:

b2fa5462 b2fbd671 b2fc8283 b2febf5e b2fed0dd b2fedd29 b2ff416e 
dfaa5f66 dfaa5f6e dfaa9fbb dfac54da dfac5f73 dfad2b4c dfade653

These missing files look just like what one would expect when sorting the pack files by name and then picking a subset of seven consecutive filenames. I’ve no idea how restic should be able to cause such a damage pattern. It rather looks like a filesystem problem on the external HDD. Please run chkdsk for the external HDD to make sure that the filesystem is intact.

1 Like

Hey Michael, Thanks for the reply. NTFS lost the disk index of those specific packs. probably due to a power failure when I was not present.

The chkdsk /f found these lost indexes and sucessfully corrected them! A new check found these packs and the rebuild-index re-added them. So now I can access the data again.

Currently considering setting up a raspberry pi so I can use ext4.

Have you considered a low cost UPS. I see a CyberPower 650VA UPS for $89 CDN that would probably support a desktop and external drive for sufficient time to terminate a backup and cleanly shut things down.

I’m honestly surprised that NTFS didn’t fix the disk issue automatically. After all NTFS is a journaling filesystem which should be able to automatically recover from this type of problems by replaying the filesystem journal.