Integrity check failed: Data seems to be missing

Getting Help
1

Any guidance on how I can resolve this is much appreciated.

I am running restic version 0.15.0 to backup a USB connected SSD drive to pCloud using rClone. It took over a month to complete a backup and I noticed multiple times that the backup was interrupted due to my vpn dropping out many times, and also on many occasions I shut down my computer prior to cancelling the backup task.

I ran this command just now:

restic check -r rclone:Archive:‘# BACKUP/SSD’

and here is part of the Log. I had to remove many lines of the same error “pack… not referenced in any index” due to size limitations in posting this:

using temporary cache in /var/folders/30/llvvthhn00176df77jcz35lc0000gn/T/restic-check-cache-3450188794
enter password for repository:
repository 8129104e opened (version 2, compression level auto)
created new cache in /var/folders/30/llvvthhn00176df77jcz35lc0000gn/T/restic-check-cache-3450188794
create exclusive lock for repository
load indexes
check all packs
pack 3787ac6e884ebd1a28625965d5ba1a04d86ad0d2b01f0cb1abd1f51630221ab5: unexpected file size: got 10076160, expected 18029111
pack 35970ed874fb2732ec74054f7dbe584441445d305cd36255c65459e1cf924457: unexpected file size: got 13152032, expected 17305869
pack 6ccbf5ce8be25fa31c0d4c8d52f4ff61554c94ad2cfe172bf73d96788350e42e: unexpected file size: got 10991712, expected 18028262
pack 906c79fde621f623017736308fbccaf75aec4c823f3b1cceb55792709ae6799d: unexpected file size: got 10600448, expected 17076870
pack c660234ad7165edbef8b298830dbc13738205b23431e1cf21ca42fadc4b5a158: unexpected file size: got 14525792, expected 16889793
pack 6ba92c2631900d7925949db3da068fafd99cd77c03dca0a0d1a13488fb90c7fe: unexpected file size: got 14876672, expected 17624160
pack 452a0e453db65407ce678de46524eb277adf5bb34a27e2b7dabb18633575b6a5: unexpected file size: got 14951776, expected 16846515
pack 5bdd603f3207b08393eae87441e1a71a73dee4498e534835d61fe0e5d21be17b: unexpected file size: got 6242304, expected 18295081
pack 5b99fd301f9b8027bec3b51406729ec5f2f868fb9b76811ebd142066ddec6862: unexpected file size: got 1572864, expected 17344660
pack 9083392b2c33edf71d5d3711965ec02cc1f45235ea0d70430023e98f76536a79: not referenced in any index
pack 065f4b36859bb4ba315b32f8439c03e31e0ea4ebb6559dd3e21a8f1769f83ff0: not referenced in any index
pack 31f1a2b02964d0ca35a2b7896b6cfc0fb2d8a6fed4c5f2ebc2b01ef646ae0b90: not referenced in any index
pack 32a2043b7e0eac4e4d4c2ab803433c3954fd5936d43f7cbd2d46bb9fc79e3fbf: not referenced in any index
pack 7a4dae0f1b9b5e18b32479fb5e538c8d4421b6208837ba17f70d5178d7c58b40: not referenced in any index
pack 16db1d5d1b49a62d3cec2d1c95488c2b92fe94447d3ec2bfb231f0925bb9c9c9: not referenced in any index
pack 343c65727e50c07c5911134edbe4e93bce260f3f6901f2f46a0ba5fbc4a80a16: not referenced in any index
pack 22234db94eb84b6f87bbc0213b52957a047686e27fddbe1d5ec1aa7de4c521f3: not referenced in any index
pack 1f283cae506826736ae3815cc2850c8653a52f6343bd11ce04c668db4f18d30e: not referenced in any index
pack 870a59f0f71132badda0d2b0df6ab9bb50530e92d2639238b7d01e1da43570b5: not referenced in any index
pack 708b1f19126c361181d1fff64c9c4dd84886326cee7579acae180fe5a6297ae3: not referenced in any index
pack 191c75105496e675b5a9db42257249cfacc723f2be426d343b90fefb81e68106: not referenced in any index
pack 39ea9d35c1609fca4b89b84a4d14f0885229d1234a773c332454077e51ecbbb0: not referenced in any index
pack 035903014abc717c50a80bd7bf7432c71faa379deacb12c0d291753cae09dceb: not referenced in any index
pack 04f6359081577d8c4613d5c693d7e1fa30440fd672deb0caa8c40a3690a65ac0: not referenced in any index
pack 2fa14169aec9856be161d3faa04cf1b19f33b067a069c1c1784c99ad0456f988: not referenced in any index
pack 71440218e7a91b6b53e4fcbe8537f82f90e2d3d240cff8fc9813daafcb542d01: not referenced in any index
pack d65e7150ceb71711dac4156d34d4fbc1498084b62c09d047c187039e84bbe153: not referenced in any index
pack 9aa8265c2f700ce55ae44226e5be3c012886e498c56adc2079c0c319c3925608: not referenced in any index
pack 7889541222e51381e6e8a153a3cdf0596115be9c28c227741db49b8e2944d3f4: not referenced in any index
pack 50b7e5bab36e6369a5aa36dd8d3bcd2f56d67a5354fbe643c72b092259e7120e: not referenced in any index
pack c3a0b46335ba85a2072d19be7c14b1540124fbcab502eb63c6d78191618dfdac: not referenced in any index
pack 0f880ae3c3917bd81b34835b6d788feb952096013bdda9ba0cc06851fee6690c: not referenced in any index
pack 9d87bf47e793eea743aec4e4271aff2fc2004095ec4acfb73f55f3dff6e8faf1: not referenced in any index
pack b71512e95ec940a16c6ce63abbeb47e80a829817db447e7a3c0ac80376b016ec: not referenced in any index
pack 158014f6bf2ac9592bcaf051dcb17c03ed476562092357bac50587652a02311a: not referenced in any index
pack c01c9ca9975bebff439a4f2f821be67c61a2284b83ee3b20573501234f4a6850: not referenced in any index
pack c568c22f03b126d22695b49ce5de5d83e1b825ab51cb3e7be3b8bf8d1b273ff1: not referenced in any index
pack fffc959cc612cd88933452293fad92c45ae0fb9d7f462e1818d8f2efa9213f9c: not referenced in any index
pack 7267d11f45ed6ed21891121b58f4bf5e346b5faba3580858f15cca91c58798dd: not referenced in any index
pack 991d5c98731263d817a43231742e46314efbc73fd48a061026c8e3d04c40a22b: not referenced in any index
pack 40df393cda474f3fde9308474e3739c745bc2952f18deaf68ee2d0609d5d188d: not referenced in any index
pack cf1373f09861ad9de7c5ae75a4c1ffb525a42ae6421be13d6e57f7045ca77c7e: not referenced in any index
pack 08b61454570918003752a7a4da145fff6c30c136a925f0333cf79a5f4e638034: not referenced in any index
pack f715dc14631e934e085bd458609475eaecc3a10d170c97ce2b3e4ab72290fb7b: not referenced in any index
pack f16c70aa63b9373fa4022ff20d611e3f9aa51b32819cc64a96ea3d02b35c1413: not referenced in any index
pack b87f74eba90ec8cade3d73827d58eec304b6564ff038c1c5ba6c6dcc02cfa30a: not referenced in any index
pack 2df12ce345ac986652bb1a78306b3c568838ae28ca9d74eaeab25e231eb05822: not referenced in any index
pack 3f27b92f4a6138e3be7d6f77dea1cb0587cb8174d7da48e6a2ac85286580e1e7: not referenced in any index
pack 82cb418b1acabb547cc1eec02990f7b0ed56bb85da3d3816d9ced434abbf490f: not referenced in any index
pack d9edeaa51c75c5697da7d0256b2f59879d712d896e3b60f68a864864337600c8: not referenced in any index
pack f3b0d9cb26d8103352323e7710325899d0830f4c7f13c51dbca2c1ca56a6de33: not referenced in any index
pack 9d0644721594c2ecad3586c83572fba5cbfb8714c1bed7ffb78fd62c7d2e059c: not referenced in any index
pack 0a85772058d761427c94340e482ff412932231a94fcc262b67e585d3882e31a0: not referenced in any index
pack d9ccc3d41725d98fd8a4604bf024e55d24b4625a735f1e5816507c663584d7d8: not referenced in any index
pack 3bb930a93ed9d3ea4d4173d2734244fe3a5d5fd66b1ac7c2a5aa69b1822059b1: not referenced in any index
pack bd2bad40c810dcb0f4ef1be122378e97824644612548667e4f842dff043e2fba: not referenced in any index

Hundreds more entries like the above “pack… not referenced in any index” which I had to emit from this post due to size limitations

609 additional files were found in the repo, which likely contain duplicate data.
This is non-critical, you can run restic prune to correct this.
check snapshots, trees and blobs
error for tree a66bee6a:
tree a66bee6a80a9801ea189c23edc8bb7dc6ff14ddb076b357a77b58bff4d778525: file “22-11-06 Q & A ｜ Counter-Trend Trades, Why 5 steps？, Examples and MORE - .mp4” blob a23aef284c517b9968fcba251e14bcbc4274665a1b84dadc75f088a20b7c879a not found in index
tree a66bee6a80a9801ea189c23edc8bb7dc6ff14ddb076b357a77b58bff4d778525: file “22-11-06 Q & A ｜ Counter-Trend Trades, Why 5 steps？, Examples and MORE - .mp4” blob 27e56a45fb0b224732bddf168da96a478fac39f4f6f0743614773a3fe5584254 not found in index
tree a66bee6a80a9801ea189c23edc8bb7dc6ff14ddb076b357a77b58bff4d778525: file “22-11-06 Q & A ｜ Counter-Trend Trades, Why 5 steps？, Examples and MORE - .mp4” blob 9b3a66c8f295f0330f4a599292571eb9fd67ec78422386275633a21e199f65ec not found in index
tree a66bee6a80a9801ea189c23edc8bb7dc6ff14ddb076b357a77b58bff4d778525: file “22-11-06 Q & A ｜ Counter-Trend Trades, Why 5 steps？, Examples and MORE - .mp4” blob 5a4e860211bf4234b6c6284108d74461ec755238674e7dcbf2af77855f2f8c67 not found in index
tree a66bee6a80a9801ea189c23edc8bb7dc6ff14ddb076b357a77b58bff4d778525: file “22-11-06 Q & A ｜ Counter-Trend Trades, Why 5 steps？, Examples and MORE - .mp4” blob e7638b66103a2cd88cc47aa0b6c52f03e4c25998d7b8ecc01c356e6ab4fb7a63 not found in index
tree a66bee6a80a9801ea189c23edc8bb7dc6ff14ddb076b357a77b58bff4d778525: file “22-11-06 Q & A ｜ Counter-Trend Trades, Why 5 steps？, Examples and MORE - .mp4” blob 836bc5d5d40d969e51fbd9f87087edec8da0d09ebd4b4ddf3bbdbdb184962c20 not found in index
tree a66bee6a80a9801ea189c23edc8bb7dc6ff14ddb076b357a77b58bff4d778525: file “22-11-06 Q & A ｜ Counter-Trend Trades, Why 5 steps？, Examples and MORE - .mp4” blob 657acff2db0a995e31ed901b87e611b25f90ccbcfd42af0ecf8536c809bb6487 not found in index
tree a66bee6a80a9801ea189c23edc8bb7dc6ff14ddb076b357a77b58bff4d778525: file “22-11-06 Q & A ｜ Counter-Trend Trades, Why 5 steps？, Examples and MORE - .mp4” blob 8014675f37bc985c7895ee909bdebda2ae373be9d9dadd878e272a0a496ec0b9 not found in index
tree a66bee6a80a9801ea189c23edc8bb7dc6ff14ddb076b357a77b58bff4d778525: file “22-11-06 Q & A ｜ Counter-Trend Trades, Why 5 steps？, Examples and MORE - .mp4” blob 8e86b14f5e2291b5901b9d0b176de409ae68c938ea21e985f61c8cf6d7323bfc not found in index
tree a66bee6a80a9801ea189c23edc8bb7dc6ff14ddb076b357a77b58bff4d778525: file “22-11-06 Q & A ｜ Counter-Trend Trades, Why 5 steps？, Examples and MORE - .mp4” blob a1a2facf9c389fd5ccf82cc4358134150a438163119ea20251524e1d15c6933d not found in index
tree a66bee6a80a9801ea189c23edc8bb7dc6ff14ddb076b357a77b58bff4d778525: file “22-11-06 Q & A ｜ Counter-Trend Trades, Why 5 steps？, Examples and MORE - .mp4” blob 1290a08deab63d33f19c7649c495481aaba7f313a01615675842f1cc46fe1538 not found in index
tree a66bee6a80a9801ea189c23edc8bb7dc6ff14ddb076b357a77b58bff4d778525: file “22-11-06 Q & A ｜ Counter-Trend Trades, Why 5 steps？, Examples and MORE - .mp4” blob fdf5dc09a09715bed1eabc47a293e95cad3b675194d3ce5146b75c5a827bfbb3 not found in index
tree a66bee6a80a9801ea189c23edc8bb7dc6ff14ddb076b357a77b58bff4d778525: file “22-11-06 Q & A ｜ Counter-Trend Trades, Why 5 steps？, Examples and MORE - .mp4” blob fe0de1bc474496d00a9ed27dc543d16d85615a9eb72cb65fb6441313bb8f4c71 not found in index
tree a66bee6a80a9801ea189c23edc8bb7dc6ff14ddb076b357a77b58bff4d778525: file “22-11-06 Q & A ｜ Counter-Trend Trades, Why 5 steps？, Examples and MORE - .mp4” blob 1b6cfe34e5e4970f868c97f2fd9583e585709307a10af61bbca6fa919b48c5da not found in index
tree a66bee6a80a9801ea189c23edc8bb7dc6ff14ddb076b357a77b58bff4d778525: file “22-11-06 Q & A ｜ Counter-Trend Trades, Why 5 steps？, Examples and MORE - .mp4” blob 7fe72899eb94831ccbcef29a38f8edb1ea7ed9d7395db512a6fe745eb6ebe5d6 not found in index
tree a66bee6a80a9801ea189c23edc8bb7dc6ff14ddb076b357a77b58bff4d778525: file “22-11-06 Q & A ｜ Counter-Trend Trades, Why 5 steps？, Examples and MORE - .mp4” blob 41739709e75c2d4008a8591160a554c987163992fb9c572df2066888a778a330 not found in index
tree a66bee6a80a9801ea189c23edc8bb7dc6ff14ddb076b357a77b58bff4d778525: file “22-11-06 Q & A ｜ Counter-Trend Trades, Why 5 steps？, Examples and MORE - .mp4” blob 79e0b0007fa9cf64bd1177ffa01b9de2ff1cace0b438b71b9cfbcc7181515545 not found in index
tree a66bee6a80a9801ea189c23edc8bb7dc6ff14ddb076b357a77b58bff4d778525: file “22-11-06 Q & A ｜ Counter-Trend Trades, Why 5 steps？, Examples and MORE - .mp4” blob 51e2ef668e5d883b8074c7d01ee3580043a1764fa2b8b85612bef2cc5d8c24b2 not found in index
tree a66bee6a80a9801ea189c23edc8bb7dc6ff14ddb076b357a77b58bff4d778525: file “22-11-13 So many opportunities, why did you not take them？ ｜ Week Analysis - .mp4” blob 30e499c5675a2941214fbf187013e829d4145c5dd16d759f7cb8fb2a857b4019 not found in index
tree a66bee6a80a9801ea189c23edc8bb7dc6ff14ddb076b357a77b58bff4d778525: file “22-11-13 So many opportunities, why did you not take them？ ｜ Week Analysis - .mp4” blob 2d728f21f60d1843d31cf0c50390539e0feebae766a16ba0776bfb99d3a68886 not found in index
tree a66bee6a80a9801ea189c23edc8bb7dc6ff14ddb076b357a77b58bff4d778525: file “22-11-13 So many opportunities, why did you not take them？ ｜ Week Analysis - .mp4” blob 68dc45a0c8189ae4dcb7904bdb70d20101f3e80a3e13f602b3806ffc361db7b4 not found in index
tree a66bee6a80a9801ea189c23edc8bb7dc6ff14ddb076b357a77b58bff4d778525: file “22-11-13 So many opportunities, why did you not take them？ ｜ Week Analysis - .mp4” blob 771633b75dbd430fe4e9769fda4294e0d68b2305f03671b7986b83c90aecf1e1 not found in index
tree a66bee6a80a9801ea189c23edc8bb7dc6ff14ddb076b357a77b58bff4d778525: file “22-11-13 So many opportunities, why did you not take them？ ｜ Week Analysis - .mp4” blob 8644af724ff66bd382ce292b54d2494e62c130502282dadea9e8a5fa095b35c0 not found in index
tree a66bee6a80a9801ea189c23edc8bb7dc6ff14ddb076b357a77b58bff4d778525: file “22-11-13 So many opportunities, why did you not take them？ ｜ Week Analysis - .mp4” blob 53812a06aa721ff30d62c303d969e6c1ad4fc65ede0538381a1381833c8df2d5 not found in index
tree a66bee6a80a9801ea189c23edc8bb7dc6ff14ddb076b357a77b58bff4d778525: file “22-11-13 So many opportunities, why did you not take them？ ｜ Week Analysis - .mp4” blob 875a785f763740eb24f112a16ffe732afe4928ce49ab573db55d3c20d736e1ec not found in index
[0:35] 100.00% 4 / 4 snapshots
Fatal: repository contains errors

I also ran:

restic prune -r rclone:Archive:‘# BACKUP/1TB SSD’

And here is the log:

enter password for repository:
repository 8129104e opened (version 2, compression level auto)
loading indexes…
loading all snapshots…
finding data that is still in use for 5 snapshots
[0:00] 100.00% 5 / 5 snapshots
searching used packs…
{<data/1290a08d> <data/1b6cfe34> <data/27e56a45> <data/2d728f21> <data/30e499c5> <data/41739709> <data/51e2ef66> <data/53812a06> <data/5a4e8602> <data/657acff2> <data/68dc45a0> <data/771633b7> <data/79e0b000> <data/7fe72899> <data/8014675f> <data/836bc5d5> <data/8644af72> <data/875a785f> <data/8e86b14f> <data/9b3a66c8> <data/a1a2facf> <data/a23aef28> <data/e7638b66> <data/fdf5dc09> <data/fe0de1bc>} not found in the index

Integrity check failed: Data seems to be missing.
Will not start prune to prevent (additional) data loss!
Please report this error (along with the output of the ‘prune’ run) at

Fatal: index is not complete

2

These warnings are caused by the interrupted backup runs, but can otherwise be safely ignored.

These files are the actual problem. Please check for those files whether there’s a second file version with the expected file size. Apparently pCloud in some cases ends up storing incomplete files even though restic got the confirmation for a successful upload.

If it is not possible to recover the files with the wrong file size, then just move those out of the repository and create a copy of the index/ folder. Afterwards run rebuild-index to fix the index and run backup again. If the backed up files haven’t changed, that should allow restic to recover missing data and repair the repository.

3

Sorry I’m new to this… how do I find if there is a second file version ? Could you give me an example command(s) to use please?

4

I’m not using pCloud myself, maybe someone can help? From what I can tell the web interface should offer you the option to access different file versions. For 3787ac6e884ebd1a28625965d5ba1a04d86ad0d2b01f0cb1abd1f51630221ab5 you’d have to look for data/37/3787ac6e884ebd1a28625965d5ba1a04d86ad0d2b01f0cb1abd1f51630221ab5 (the intermediate folder name are the first two characters of the file name)

5

Thankyou… I managed to view the file versions using the web interface. And yes there is a second file version with the expected size. So should I just assign this as the current and correct revision? I can see the option to do this in the web interface…

Then after I have done this with all the “unexpected file size” errors, what do I need to do if anything ?

Thanks again!!

6

Yes please.

Run check again to make sure your repository has been repaired successfully.

7

Thankyou… I have run check again and the “unexpected file size” errors have disappeared, however the errors “not found in index” are still appearing as per the log provided originally… how to proceed from here ?

Thanks again!

8

Did you run rebuild-index? To repair the repository, the steps from Route 2 of Recover from broken pack file · Issue #828 · restic/restic · GitHub should work.

9

I’ve got the exact same issue as the OP.

loading indexes...
loading all snapshots...
finding data that is still in use for 1426 snapshots
[1:27] 100.00%  1426 / 1426 snapshots

searching used packs...
{<data/1e61779b> <data/30a6fdcd> <data/398d419c> <data/6cacc19e> <data/790b051d> <data/829aeeca> <data/9f0cbd08> <data/b422c7d8> <data/c30fb622> <data/d2517b3a> <data/d86e2781> <data/e6153204> <data/ea7adef9> <data/f81face4>} not found in the index

Integrity check failed: Data seems to be missing.
Will not start prune to prevent (additional) data loss!
Please report this error (along with the output of the 'prune' run) at
https://github.com/restic/restic/issues/new/choose
Fatal: index is not complete

I have ran the restic check command to find all of the snapshots, and the list I got was every single snapshot - so I cannot delete the snapshots.

I then ran the restic repair index + repair snapshots command after getting the dev version of Restic. Then I rebuilt the index, backed up again, then ran the prune command. But I was met with the exact same error, 1:1.

There are only a few files showing up in Restic find, and I am okay with deleting them. How can I repair my repo?

Edit: I forgot add --forget to restic repair snapshots --forget. I have re-ran it with --forget, and now when checking the repo, no errors are found.

Edit 2: I ran check again, but this time with the read data argument. I now get failures saying:

read all data
Pack ID does not match, want 93856ad0e9d093780a518f5423ecd7da52e155d7f9bb10ee1b3a30cc66337de1, got 46c174142f8d30f3d787b0addb3d4df7c8620768d27d6ce9491ea4521babed2e
Pack ID does not match, want c5ca0a01e3879324a0f804270a178f8fdda120fa995a5098506deb9d57fe7e22, got 8a58b6997b6daef91e72c720365ef41f5e05cfc7692f8fc019dca9b489b4b54b
Pack ID does not match, want fa9cea6f583f336d26f5497a5bf3676cada7b48f13ab469ae6034cda7de6113f, got 4d97809709738cba6326bf395c40c307574b8334301bfe216bf489d2dd856841
Pack ID does not match, want da90a2e2a6eb8293e51ac76ad52e616281ff6ba4e7e151a6a453f5d72fd31daf, got e5c399c7c68d421fb6644cc186d21fad2b3de2c9bdc0dcf482f572977ede27fd
[7:47:13] 100.00% 41005 / 41005 packs

Fatal: repository contains errors

10

I can’t edit my message above. For the second edit which mentions the check with the read all argument failing, that was a false alarm. I checked the wrong log. The latest check with the read all argument has passed.

11

Then prune should work now?

There’s still the question how the repository was damaged in the first place. Which restic version did you use to create your backups? Which backend?