Damaged repository with ciphertext verification failed

Getting Help
1

Hello,

I’ve encountered an error using restic while trying to create a backup to an external drive, after checking the repository it looks like I smashed the drive against the wall, repeatedly.

Software versions

Restic

I have two restic versions because 0.11.0 is from my distro and the other manually downloaded to the latest version, it has been called 123restic to avoid colliding with the native one.

❯ restic version
restic 0.11.0 compiled with go1.15.9 on linux/amd64
❯ 123restic version
restic 0.17.1 compiled with go1.23.1 on linux/amd64

Operating system

Debian GNU/Linux 11 (bullseye)

Kernel

Linux lilchewchew2-0 5.10.0-32-amd64 #1 SMP Debian 5.10.223-1 (2024-08-10) x86_64 GNU/Linux

ZFS status

root@lilchewchew2-0:~# zpool status
  pool: gringotts
 state: ONLINE
  scan: scrub repaired 0B in 16:27:56 with 0 errors on Tue Oct  1 16:27:56 2024
config:

	NAME                                 STATE     READ WRITE CKSUM
	gringotts                            ONLINE       0     0     0
	 raidz2-0                           ONLINE       0     0     0
	   ata-ST8000VN004-2M2101_WSD3C3KH  ONLINE       0     0     0
	   ata-ST8000VN004-2M2101_WSD3PWW3  ONLINE       0     0     0
	   ata-ST8000VN004-2M2101_WKD31SR3  ONLINE       0     0     0
	   ata-ST8000NE001-2M7101_WSD42629  ONLINE       0     0     0
	   ata-ST8000VN004-2M2101_WSD46GRJ  ONLINE       0     0     0
	   ata-ST8000VN004-2M2101_WSD1HAVC  ONLINE       0     0     0

errors: No known data errors

Description of problem

I’ve been using restic 0.11.0 for about three months trying to do weekly backups to two different external drives just to see if restic is the solution to my backup needs.

It might be worth mentioning that at some point in time I mounted this repo just to check I could access the files, nothing bad happened.

It was all perfect until one fateful day last week the backup failed and restic check would only vaguely mention input/output errors.

I usually do my backups this way:

nohup restic -r /mnt/alpha/alpha-backup --password-file restic-password.txt --verbose backup /gringotts/NAS /gringotts/ON /gringotts/MH 2>&1 > restic-alpha-incremental.log &

After the backup failure I ran:

restic -r /mnt/alpha/alpha-backup --password-file restic-password.txt check | tee restic-check.log

Which I don’t have the output of anymore, but didn’t give me too much information.

Next I tried to do restic check --read-data but cancelled it because it was taking too long and it was showing input/output errors and I thought that my drive was dying. I had to unlock the repository after this.

Very long story short, after enabling SMART on my external drive through usb-storage quirks, I ran a conveyance and a long test on my drive fully expecting the Reallocated_Sector_Ct to skyrocket but it came out fine. I was still sure the drive must be dying to so I ran two sets of badblocks (one RO and the other RW non-destructive) and my SMART stats still looked fine, this is when I started suspecting the restic version.

After this I tried to fix it by running a prune as the log suggested but it would also show input/output errors and do nothing, so I stopped completely.

From this point onwards I start using restic 0.17.1

I thought sane to execute restic check --read-data and capture the log to see if the new version catches something different (as advised in the wiki), the check was executed with this command:

123restic -r /mnt/alpha/alpha-backup --password-file restic-password.txt check --read-data 2>&1 | tee restic-check-read.log

And these are bits and pieces of the log because it’s huge and repeating:

using temporary cache in /tmp/restic-check-cache-3502872482
create exclusive lock for repository
load indexes
check all packs
List(data) returned error, retrying after 1.459542456s: readdirent /mnt/alpha/alpha-backup/data/bf: input/output error
List(data) returned error, retrying after 2.243790173s: readdirent /mnt/alpha/alpha-backup/data/bf: input/output error
List(data) returned error, retrying after 3.151110364s: readdirent /mnt/alpha/alpha-backup/data/bf: input/output error
List(data) returned error, retrying after 6.814500226s: readdirent /mnt/alpha/alpha-backup/data/bf: input/output error
List(data) returned error, retrying after 20.931240592s: readdirent /mnt/alpha/alpha-backup/data/bf: input/output error
List(data) returned error, retrying after 33.79845222s: readdirent /mnt/alpha/alpha-backup/data/bf: input/output error
List(data) returned error, retrying after 39.209403935s: readdirent /mnt/alpha/alpha-backup/data/bf: input/output error
List(data) returned error, retrying after 1m9.380641299s: readdirent /mnt/alpha/alpha-backup/data/bf: input/output error
List(data) returned error, retrying after 1m29.378931155s: readdirent /mnt/alpha/alpha-backup/data/bf: input/output error
List(data) returned error, retrying after 1m14.570235105s: readdirent /mnt/alpha/alpha-backup/data/bf: input/output error
List(data) returned error, retrying after 30.750545292s: readdirent /mnt/alpha/alpha-backup/data/bf: input/output error
List(data) returned error, retrying after 1m23.030758317s: readdirent /mnt/alpha/alpha-backup/data/bf: input/output error
List(data) returned error, retrying after 37.947299167s: readdirent /mnt/alpha/alpha-backup/data/bf: input/output error
List(data) returned error, retrying after 50.852579432s: readdirent /mnt/alpha/alpha-backup/data/bf: input/output error
List(data) returned error, retrying after 1m8.779811325s: readdirent /mnt/alpha/alpha-backup/data/bf: input/output error
List(data) returned error, retrying after 49.139775076s: readdirent /mnt/alpha/alpha-backup/data/bf: input/output error
List(data) returned error, retrying after 1m18.463066226s: readdirent /mnt/alpha/alpha-backup/data/bf: input/output error
List(data) returned error, retrying after 1m0.104003864s: readdirent /mnt/alpha/alpha-backup/data/bf: input/output error
List(data) failed: readdirent /mnt/alpha/alpha-backup/data/bf: input/output error
readdirent /mnt/alpha/alpha-backup/data/bf: input/output error
pack cbb46f6c11eee2f4b9b9f934ecd5bf4b56db0ca553dc717352dda18126d18e80: does not exist
pack c069376cadfda5901e638c49e2240103fc593bfbc5d6072ba4bf18658be7da22: does not exist
pack ef97ce9013d043ff848e03eb65d30c90a02464d5f452e01e0157f0bbe7185274: does not exist
pack fa7cb4ff5cca47530804cfba5f5f112721ef0a126f2e9f1eca6f54d3f0052a25: does not exist
pack c92f4aedacc9b7b5c6038b7a8c1b79d896a6f49c682ce155c4efa986b0939160: does not exist
pack fdabb7e4c7c967892cf6239c7e85a0da08178dd735e3b8293fa3df479dd53fb4: does not exist

<this repeats forever with different packs>

check snapshots, trees and blobs
[0:01] 100.00%  13 / 13 snapshots
read all data
pack d0b0d0de9042b7be668ef687f46207b8fe917157e839c2d00d7769fd162e1ae0 contains 2 errors: [blob f69c51bbd3adf21663c0c3da7e47ea5b6c1cc3538c14105101c0d67f4ddfaa7c: decrypting blob <data/f69c51bb> from d0b0d0de failed: ciphertext verification failed unexpected pack id 536ca2e9f8b98da0a9900621f6ffc2f8d0037d0a7524ccc33574c20d9a6fdbce]
pack 71923498dfb15926158e06705726517d829a9493a5c5a90e8abbe7c135bd7b31 contains 2 errors: [blob a96d17b7ff2b325f060ec47f3781f5fd2914499632aa540d52c5c8135517532c: decrypting blob <data/a96d17b7> from 71923498 failed: ciphertext verification failed unexpected pack id 3980629925e63046bba1b3a81a7055044d639d5c7f923a4271ee3e20a07dff8c]
pack 3191ef8002a19dc8790e88e5dba8130d3726fe0215cc36f2a33068cc6cb950eb contains 2 errors: [blob 0748a4bf40a1a001cd933e2f8a37b0eb80fef29358b6d4def70d0b0e9b94ab31: decrypting blob <data/0748a4bf> from 3191ef80 failed: ciphertext verification failed unexpected pack id 732ebc641c73a55814110f31f23a86a1aaea417292727eaab101abdc185692fb]
pack 32c369f62640df226429b365d2e25f73f8ad2e22d16de8d4be69be37509fb1c7 contains 2 errors: [blob 3dddb4ca74459ebd3be93bfb3dde2086188d5897e7ff8e8debc527f2a4618a66: decrypting blob <data/3dddb4ca> from 32c369f6 failed: nonce is invalid unexpected pack id 479e32217520e7c7b79a88973d13f92fc2b8b58be21ec580876dd594e397b365]
pack 52d8613387a99671f3cb44f04db51914cd5ee100b077065b94c3b587414fe8f1 contains 2 errors: [blob e6fbeff2ac7d2efd7c9976a291fb4ab2d71f71669e0101ee3d530ead85a58613: decrypting blob <data/e6fbeff2> from 52d86133 failed: ciphertext verification failed unexpected pack id ce823dcf61e1841ea576007700dd16e4b74cc39cf0003d16ed00c242fb571d60]
pack 43c19e0d42730145e5463d643b1fc16ce175004c4c12f2913dd3a28c9cbe90dd contains 2 errors: [blob 0ce72796d4c99b52f7f2a03f0ee86a01a32d4d563c0c778229fab63836d3a6c3: decrypting blob <data/0ce72796> from 43c19e0d failed: ciphertext verification failed unexpected pack id e610da91bbf0115867813b0a14b540316dec7a5096bd0935682e0ac33b23126c]
pack f3ea14cd7332ecd1c9abb556bb4fab6339ad7213f82fa40240acf081175d4e16 contains 2 errors: [blob 14ec8230727fba63f6c03cbcefdb4b1e99452e203e19edaa01fd7be2518886e9: decrypting blob <data/14ec8230> from f3ea14cd failed: ciphertext verification failed unexpected pack id 3ad8e0c0e591d21423a0ab0c927686ffcb931753c5dbfb9ba15f8fc79b4a61a0]
pack a3c5824930c11f4ade412a49995d465cf434e1379d0a0a7adfb4e8ef1ef6b886 contains 2 errors: [blob bc69b0dab67e890a59d3376a870a950ac94a5e27cdc3d22c982c480a165b2bf9: decrypting blob <data/bc69b0da> from a3c58249 failed: ciphertext verification failed unexpected pack id 2c40edb626fc70766521b4db42ddfe90bf6ea43939ca85d0b293dfeb5b51fb2c]
pack c33c747ab0a73fdf5c79c4a1403d960afedb84b5745e7b0d8fe4658f7223da28 contains 2 errors: [blob 3207e176983edc9af981f551caac72b1d6041b7bc57e96e03e16c528cc891aad: decrypting blob <data/3207e176> from c33c747a failed: ciphertext verification failed unexpected pack id 1781f468366cccb7694ae05436f83f922f981879b3bf93bec1ac595791e9de3b]
pack 34b15eb1dc61fea1b674f93f1b081ac6de94ae67c365d0f9534d66e3b4c7a1a1 contains 2 errors: [blob 67cb7560417a1e57fd2435cc65c25c9374dff0dc490afbf128eb4abf2dd72e27: decrypting blob <data/67cb7560> from 34b15eb1 failed: ciphertext verification failed unexpected pack id 2fa2f4c6f6ef8a680cddb2c0a351ea05de2546def6244feb81a8187c9ece7894]
pack 05df8e387f2804f7f4e7ba28e6c6ab8408b2a51ac34c6347d0e79fc5867b6623 contains 2 errors: [blob 86e8679f7129e1e3b96fcc74a13b0eef7adb501422ece138fde52883cc74ee23: decrypting blob <data/86e8679f> from 05df8e38 failed: ciphertext verification failed unexpected pack id 105a1f0546213701482a102b460ef98a1ce5ca9950c92448b8dbc223d855ac92]
pack 15aeaf95d3ded827ae1d5b55b21e6363cad6d67cc81c70e9a14ca0b2c950729d contains 2 errors: [blob b9546de7716b5880ba06c02967ef52ed2fa3b81aebc22f413051f126b8583197: decrypting blob <data/b9546de7> from 15aeaf95 failed: ciphertext verification failed unexpected pack id dfd65f3089f4a9483f0cf04475b094f0f0cd6661ab6704654eeb9c34187e6375]
pack c5037995c681f437d311a89e6e3ab9f3115e395953f631d1f9038079284a4c5c contains 2 errors: [blob b171853902d4a98c97eb60f8db96d544172bc76409bc16b1a988c53f537185e9: decrypting blob <data/b1718539> from c5037995 failed: ciphertext verification failed unexpected pack id 1d678e94b1671c47003bce489750739da111dbb6e0af32933a1faf8ac36197aa]
pack 15ed3deb342d5ab683fea7577b49b76e5ed1dc1098955755af8ebb59831f8b53 contains 2 errors: [blob c2d670c596c78443875529fd29912980470c20f99f224b55d6e5cad455b2c8a7: decrypting blob <data/c2d670c5> from 15ed3deb failed: ciphertext verification failed unexpected pack id 33998ad7f75b862366c5fb4687fd8803ecdfd1ddd7a9485d6dab6f5fd6fff9d5]
pack c6cd7efe7193b6ef41d7007a34f12262b6f9f9f3f34b8ecd541e5a8995fe1eb5 contains 2 errors: [blob 331a44e3a225c51440467c967cf0fecea898724d86d2ed4b56e22b46cb72d019: decrypting blob <data/331a44e3> from c6cd7efe failed: ciphertext verification failed unexpected pack id 41246b97f0f3f3d80afdcbde3b0ba7c0b6c6dafc9615de463c85421dc7be05fc]
pack 067803e995685a4b1516add892c7dad7a00e6fc15c3323ae79c94f809a357297 contains 2 errors: [blob c5d9059f69a622875850adcd02aa53ca37643384cb4593ffc09cb8d21feda6d2: decrypting blob <data/c5d9059f> from 067803e9 failed: ciphertext verification failed unexpected pack id cf0d8840d9d962c1f793a6aace086a6be5a40ebb24330a1f05726eded94955ce]
pack 16a81b3e7affb66ed02490eec34386ab087a01b593ece61922200fd55eba4297 contains 2 errors: [blob 5899f49acd08c4c126905adc8885702e66522342b709047bac300d8bede83b26: decrypting blob <data/5899f49a> from 16a81b3e failed: ciphertext verification failed unexpected pack id 7a3a5c3676797795b7e9365c4512f3b9a9eb3f1625ba637749733938d0178a31]
pack 161484e63a65d345681ce7e9334c63289928381250bf45c7956aa741315d9f58 contains 2 errors: [blob da0c4312d8d2d4ffbd8b97fec609ae5ece29889e2c1a276e65de686f3befaf16: decrypting blob <data/da0c4312> from 161484e6 failed: ciphertext verification failed unexpected pack id e89b7daf827e561960a1892a9b8cdd9e71fb8443520d3515e59f6c78758b9c65]
pack 679fccd2fd59e73bc13f7c6f2aab2ec3ad2b8748c19d9a39cd93c7735cef4a1e contains 2 errors: [blob 63f384b182ee0946b3a0729d58d2ae1d917e36d25b2945b6a3703ae8b975a6c6: decrypting blob <data/63f384b1> from 679fccd2 failed: ciphertext verification failed unexpected pack id e40a32b8e69fa8f81b13e121c67dec8d8a10306536694d87f205c89df7f99088]
pack b88225fae4749651fb7eae99a5c5477f178c74e9a0187f498a58fcad1084a4a9 contains 2 errors: [blob c722e8bc16acc10d66e797b57a21162bf2fba93ab573e1bf9dfd82cf600fb57f: decrypting blob <data/c722e8bc> from b88225fa failed: ciphertext verification failed unexpected pack id 93a9352e1179c3fbe633bc8dcdf2efee548b7c391515c58ed6e1cc4517fd0f78]
pack 0837357f89698c36a3e78279d7ef212d9151cd4a91338705f4583c53f1f10ad9 contains 2 errors: [blob 1f3bc4912ed7962c94dfa7e9d38609bfda32e533be9f4ddacada477600bce0cb: decrypting blob <data/1f3bc491> from 0837357f failed: ciphertext verification failed unexpected pack id b2cefe8ac14b653e4a4ff59e3ef243c5f9e2a50e6ad5b8e2524dc39662191d54]
pack b8cf64cd61e59a9fd2fd12eea45e3765dc7f519a27d3eac2bebbeaaac601b704 contains 2 errors: [blob d2b11592ed3d772d7b9f8e21407878e961bada4bc34154684e69728f5f69270e: decrypting blob <data/d2b11592> from b8cf64cd failed: ciphertext verification failed unexpected pack id aeaa8f5c1e35e72b33edf41f116fa1c03a1c2d15c22e6ba0e1a692807f334018]
pack c90da86133c5e3678cefbd50a4ffa14d92a1a3d3e1821ad89e9d95d1a2790fbc contains 2 errors: [blob 1f0d6926430600a3cd2ae6f9e669650ac3ba5879e701de0cac9cf9b220938ade: decrypting blob <data/1f0d6926> from c90da861 failed: ciphertext verification failed unexpected pack id 665fd1cf3066fe230c14b0eb1e6fd3c57a2d5182662dad152cab06e01cd1db8a]
pack 1b07b57b52391e630792aaf85dc13dc6086bc8f776c22f7b5e4a86c7c5d73aea contains 2 errors: [blob aafbb6e7299ac6e69da06fef47f9cfe2dacbb1f7dfd8c95a357bdd10770560f3: decrypting blob <data/aafbb6e7> from 1b07b57b failed: ciphertext verification failed unexpected pack id f880dbce0e14fd4a10f3d13a956ca61257337cefddd4b017f55a2c62a2429f0a]
pack 5b5e0692d2d828caa35c84d1d7fdd7743afee69a5eeb97fc03b49c76eb10abef contains 2 errors: [blob 6aaf40065d49c857cfa725df59de058d38924449f0dcb311ed0682741a39f649: decrypting blob <data/6aaf4006> from 5b5e0692 failed: ciphertext verification failed unexpected pack id ec8537b9cd67a0ff55e88240f3bdf48fba80e633994f2334fc5ebfe701ed4a8b]
pack 7ccfa4fbc681a1f81e25706e2fc01b7fa24679532ba42ed5e769b07af2532703 contains 2 errors: [blob 7299f09abe66f430cd98380177c4b8bf00269a15629cd0645834d26b4c927103: decrypting blob <data/7299f09a> from 7ccfa4fb failed: ciphertext verification failed unexpected pack id 5fafd762e4f302b0c8147861367915f15fcda5cad2472ab7a1ebc283b258b59e]
pack 1cc6e43b5ba4050ed723e5319533e9b1fa5349b1f715c8e00fa080ea54e2fbb7 contains 2 errors: [blob 4904a9747948d360a6454ecf7f349c8f248a67ea9d6b6e7fcdac26ff8ef4f9d0: decrypting blob <data/4904a974> from 1cc6e43b failed: ciphertext verification failed unexpected pack id 850beb18d0dc72e371bde20c28b40ce664a3ec17663f0bc02af37b12870a84ed]
pack 4c5f4c9146f07860d61d99f5749cb3f3e1fe5f3e8de704c30c90fe76a6e46f49 contains 2 errors: [blob 82cff50ff9990efcc0900802403ecb62d8f4669a27081ebbb0afeac4b4ac7b1f: decrypting blob <data/82cff50f> from 4c5f4c91 failed: ciphertext verification failed unexpected pack id 1ed0ed686aaa9cba2f2c7463cc5bd5855c6b5aed1b1caf825d6df2ac9c7240e4]
pack 5c1d153cc57b04574538788fd13dcee7caca382c2888d4084c0ec7155d25c011 contains 2 errors: [blob 095616b34c05c7c203953f302dfa1473f78f74de4cc2346c074f6b311d39d964: decrypting blob <data/095616b3> from 5c1d153c failed: ciphertext verification failed unexpected pack id ca7088c36b8c5183da1dbae4a16bc83a687faf868b0116c68c6a23140b307eaa]
pack 8c00fe576a96b4bbce3deb4a0f755e4fc32af242252175f664c20cfc8bb35238 contains 2 errors: [blob 28afe7eb6966730936558a742649de2151ba86b00eeaca2c3173657c7b287803: decrypting blob <data/28afe7eb> from 8c00fe57 failed: ciphertext verification failed unexpected pack id 02a09df7cd808eaa8c75d81bee41c8df0261ddf2322ad204b0e2b57b07408889]
pack 4d0bee84353bf456b7455839c9cb25e667c71e367461393c59848a27dcdb5abc contains 2 errors: [blob 0b74ccc1c27781343aaf606700bf4dbbe1e4cc63fe5768ba27647903a1bf7017: decrypting blob <data/0b74ccc1> from 4d0bee84 failed: ciphertext verification failed unexpected pack id 50731774078d2089e420854195f997ff2d361a86fb04ce7610905e8fbeb90929]
pack 8eb64ec7051ca0e0bba85aad27989638881997f73cd04fd67da1f705abe4a8e8 contains 2 errors: [blob 6ab7ef8f39598599f36ddfa54d765fc1697a3fafb824319d2de5fd903b05cf03: decrypting blob <data/6ab7ef8f> from 8eb64ec7 failed: ciphertext verification failed unexpected pack id d76ab7d5c79f71f41fa05cbba401729286b016a403e9afbbf9e92898411c49df]
pack ceb51a9c1a9fa14c3f7fa91593bb7d0c07adc5bdb0c43f9701560df63070cb8b contains 2 errors: [blob ce0473cfa22230e996ff4bee5fb7ed2ee3c84e9804a07a2176be5b73ff421acc: decrypting blob <data/ce0473cf> from ceb51a9c failed: ciphertext verification failed unexpected pack id 5e28533102daa404029f905d07a6ea73b41974c8e0cc57b09fba39fdfb831fcf]
pack 7e3b98ac39c7b6aafd9c794c7460d72645138576723d622456e9b6fcfdb3f83e contains 2 errors: [blob 2d309f656ecf7c03c03e64f5b7da2b9194668cef8bc715b43f3f8fb3c42d60c1: decrypting blob <data/2d309f65> from 7e3b98ac failed: ciphertext verification failed unexpected pack id cd972f41dc26c70c3fb3a748c6f86ec5cd5b3345384664cb149f51bc06e6d5b3]
Load(<data/bf1ebf05d4>, 4754151, 0) failed: open /mnt/alpha/alpha-backup/data/bf/bf1ebf05d4871d9dcdd02a47f74535e680cb42d511368d1baf6e71c380f94712: no such file or directory
Load(<data/bf1ebf05d4>, 4754151, 0) failed: open /mnt/alpha/alpha-backup/data/bf/bf1ebf05d4871d9dcdd02a47f74535e680cb42d511368d1baf6e71c380f94712: no such file or directory
download error: open /mnt/alpha/alpha-backup/data/bf/bf1ebf05d4871d9dcdd02a47f74535e680cb42d511368d1baf6e71c380f94712: no such file or directory
Load(<data/bf229d9872>, 4801376, 0) failed: open /mnt/alpha/alpha-backup/data/bf/bf229d98728e10ab6cbccc81c4d8133cbf8f64a2a22cf01f65c340f8c445fdab: no such file or directory
Load(<data/bf229d9872>, 4801376, 0) failed: open /mnt/alpha/alpha-backup/data/bf/bf229d98728e10ab6cbccc81c4d8133cbf8f64a2a22cf01f65c340f8c445fdab: no such file or directory
download error: open /mnt/alpha/alpha-backup/data/bf/bf229d98728e10ab6cbccc81c4d8133cbf8f64a2a22cf01f65c340f8c445fdab: no such file or directory
pack afcd2309393dc03c4e5bff6539d85947aa6935a2d13a5d2fca78145e800f90fb contains 2 errors: [blob 0d49f654b435fd82faa67d3399a365c7bec27165afa0d3ca11fd46d0b62f8322: decrypting blob <data/0d49f654> from afcd2309 failed: ciphertext verification failed unexpected pack id c993346a1aa636d9da61f4a3cb6f1458c03c23caead5b01b68a129a6e4020a7a]
Load(<data/bf1cadf62e>, 4729363, 0) failed: open /mnt/alpha/alpha-backup/data/bf/bf1cadf62e0372f741dfbcf431953106c7ef217669e28f202a42423708c62076: no such file or directory
Load(<data/bf1cadf62e>, 4729363, 0) failed: open /mnt/alpha/alpha-backup/data/bf/bf1cadf62e0372f741dfbcf431953106c7ef217669e28f202a42423708c62076: no such file or directory
download error: open /mnt/alpha/alpha-backup/data/bf/bf1cadf62e0372f741dfbcf431953106c7ef217669e28f202a42423708c62076: no such file or directory
Load(<data/bf21e5ae0e>, 5297179, 0) failed: open /mnt/alpha/alpha-backup/data/bf/bf21e5ae0e0713ad7b457c2e3dac04fd8f2efdd6a8475836bed3ca2d0af45146: no such file or directory
Load(<data/bf21e5ae0e>, 5297179, 0) failed: open /mnt/alpha/alpha-backup/data/bf/bf21e5ae0e0713ad7b457c2e3dac04fd8f2efdd6a8475836bed3ca2d0af45146: no such file or directory
download error: open /mnt/alpha/alpha-backup/data/bf/bf21e5ae0e0713ad7b457c2e3dac04fd8f2efdd6a8475836bed3ca2d0af45146: no such file or directory
Load(<data/bf1c27335e>, 4729100, 0) failed: open /mnt/alpha/alpha-backup/data/bf/bf1c27335e6b19f33edb2d1bef36268ada2e1f437de5867a77b3cc430ea9a941: no such file or directory
Load(<data/bf1c27335e>, 4729100, 0) failed: open /mnt/alpha/alpha-backup/data/bf/bf1c27335e6b19f33edb2d1bef36268ada2e1f437de5867a77b3cc430ea9a941: no such file or directory
download error: open /mnt/alpha/alpha-backup/data/bf/bf1c27335e6b19f33edb2d1bef36268ada2e1f437de5867a77b3cc430ea9a941: no such file or directory
Load(<data/bf21c3286e>, 4546745, 0) failed: open /mnt/alpha/alpha-backup/data/bf/bf21c3286ef353b85de53f2d74009b1b707dee26bb188c82171dac7a94bba18f: no such file or directory
Load(<data/bf21c3286e>, 4546745, 0) failed: open /mnt/alpha/alpha-backup/data/bf/bf21c3286ef353b85de53f2d74009b1b707dee26bb188c82171dac7a94bba18f: no such file or directory
download error: open /mnt/alpha/alpha-backup/data/bf/bf21c3286ef353b85de53f2d74009b1b707dee26bb188c82171dac7a94bba18f: no such file or directory
Load(<data/bf1fc38c63>, 4808236, 0) failed: open /mnt/alpha/alpha-backup/data/bf/bf1fc38c63ee13de90ef9cf573331ee90c21296c3351b19b24c1373e51b5b766: no such file or directory
Load(<data/bf1fc38c63>, 4808236, 0) failed: open /mnt/alpha/alpha-backup/data/bf/bf1fc38c63ee13de90ef9cf573331ee90c21296c3351b19b24c1373e51b5b766: no such file or directory
download error: open /mnt/alpha/alpha-backup/data/bf/bf1fc38c63ee13de90ef9cf573331ee90c21296c3351b19b24c1373e51b5b766: no such file or directory
Load(<data/bf2028c95f>, 4450257, 0) failed: open /mnt/alpha/alpha-backup/data/bf/bf2028c95f536475fa3d4168258cf307e8c1b2e413ca00d57c9146862c802b71: no such file or directory
Load(<data/bf2028c95f>, 4450257, 0) failed: open /mnt/alpha/alpha-backup/data/bf/bf2028c95f536475fa3d4168258cf307e8c1b2e413ca00d57c9146862c802b71: no such file or directory
download error: open /mnt/alpha/alpha-backup/data/bf/bf2028c95f536475fa3d4168258cf307e8c1b2e413ca00d57c9146862c802b71: no such file or directory
pack bf14413f9d25af261ac92504e04e2b86e97ebdd4801abf15d7d0beeee1c56854 contains 2 errors: [blob f83c0a424bd57cc2233743131a8346a5b1a2878a5ed02250ca05376cdd34b8fa: decrypting blob <data/f83c0a42> from bf14413f failed: nonce is invalid unexpected pack id 565befe7ac4c9bc46ecb830e1fcde00aad7e20db8f998a83cb83f1fd42cfafae]
[6:28:53] 100.00%  81729 / 81729 packs

The repository contains damaged pack files. These damaged files must be removed to repair the repository. This can be done using the following commands. Please read the troubleshooting guide at https://restic.readthedocs.io/en/stable/077_troubleshooting.html first.

restic repair packs a3c5824930c11f4ade412a49995d465cf434e1379d0a0a7adfb4e8ef1ef6b886 1b07b57b52391e630792aaf85dc13dc6086bc8f776c22f7b5e4a86c7c5d73aea 4d0bee84353bf456b7455839c9cb25e667c71e367461393c59848a27dcdb5abc 7e3b98ac39c7b6aafd9c794c7460d72645138576723d622456e9b6fcfdb3f83e afcd2309393dc03c4e5bff6539d85947aa6935a2d13a5d2fca78145e800f90fb f3ea14cd7332ecd1c9abb556bb4fab6339ad7213f82fa40240acf081175d4e16 7ccfa4fbc681a1f81e25706e2fc01b7fa24679532ba42ed5e769b07af2532703 4c5f4c9146f07860d61d99f5749cb3f3e1fe5f3e8de704c30c90fe76a6e46f49 5c1d153cc57b04574538788fd13dcee7caca382c2888d4084c0ec7155d25c011 8eb64ec7051ca0e0bba85aad27989638881997f73cd04fd67da1f705abe4a8e8 c33c747ab0a73fdf5c79c4a1403d960afedb84b5745e7b0d8fe4658f7223da28 34b15eb1dc61fea1b674f93f1b081ac6de94ae67c365d0f9534d66e3b4c7a1a1 15aeaf95d3ded827ae1d5b55b21e6363cad6d67cc81c70e9a14ca0b2c950729d 679fccd2fd59e73bc13f7c6f2aab2ec3ad2b8748c19d9a39cd93c7735cef4a1e 1cc6e43b5ba4050ed723e5319533e9b1fa5349b1f715c8e00fa080ea54e2fbb7 32c369f62640df226429b365d2e25f73f8ad2e22d16de8d4be69be37509fb1c7 c5037995c681f437d311a89e6e3ab9f3115e395953f631d1f9038079284a4c5c 5b5e0692d2d828caa35c84d1d7fdd7743afee69a5eeb97fc03b49c76eb10abef 3191ef8002a19dc8790e88e5dba8130d3726fe0215cc36f2a33068cc6cb950eb 15ed3deb342d5ab683fea7577b49b76e5ed1dc1098955755af8ebb59831f8b53 8c00fe576a96b4bbce3deb4a0f755e4fc32af242252175f664c20cfc8bb35238 ceb51a9c1a9fa14c3f7fa91593bb7d0c07adc5bdb0c43f9701560df63070cb8b 161484e63a65d345681ce7e9334c63289928381250bf45c7956aa741315d9f58 b88225fae4749651fb7eae99a5c5477f178c74e9a0187f498a58fcad1084a4a9 0837357f89698c36a3e78279d7ef212d9151cd4a91338705f4583c53f1f10ad9 d0b0d0de9042b7be668ef687f46207b8fe917157e839c2d00d7769fd162e1ae0 05df8e387f2804f7f4e7ba28e6c6ab8408b2a51ac34c6347d0e79fc5867b6623 c6cd7efe7193b6ef41d7007a34f12262b6f9f9f3f34b8ecd541e5a8995fe1eb5 067803e995685a4b1516add892c7dad7a00e6fc15c3323ae79c94f809a357297 16a81b3e7affb66ed02490eec34386ab087a01b593ece61922200fd55eba4297 52d8613387a99671f3cb44f04db51914cd5ee100b077065b94c3b587414fe8f1 43c19e0d42730145e5463d643b1fc16ce175004c4c12f2913dd3a28c9cbe90dd 71923498dfb15926158e06705726517d829a9493a5c5a90e8abbe7c135bd7b31 b8cf64cd61e59a9fd2fd12eea45e3765dc7f519a27d3eac2bebbeaaac601b704 c90da86133c5e3678cefbd50a4ffa14d92a1a3d3e1821ad89e9d95d1a2790fbc bf14413f9d25af261ac92504e04e2b86e97ebdd4801abf15d7d0beeee1c56854
restic repair snapshots --forget

Damaged pack files can be caused by backend problems, hardware problems or bugs in restic. Please open an issue at https://github.com/restic/restic/issues/new/choose for further troubleshooting!
Fatal: repository contains errors

Moving forward

I’m not sure how I should proceed here, I could certainly try to fix the repository or just delete and recreate, or this could be a good opportunity to study these errors. I also have another drive that has something very close (a few files off) to the same repository that I have not tried to update and should be working fine.

  1. Is there anything worth investigating here or should these errors be attributed to an old version of restic and the unreliability of cables/connectors/components?
  2. Is it safe to assume that I should look away from the old restic version and stick to the newest I can get? I thought for a moment that I might be able to get away with the version Debian provided
  3. Is it worth trying to repair my repository or should I just create a new one given that the source data is fine? Or maybe I should copy the repository from my other drive instead of recreating it?

I would like to borrow your expertise on the subject to decide how to move forward in this case.

Thanks in advance!

Best regards.

2

An “input/output error” usually means a hardware failure. Could be the media (I.e. your HDD is buggered) or cabling etc. I’d attempt to reconcile the hardware issues before attempting to recover the repository.

Troubleshoot your storage and work out whether its recoverable (I.e. replace bad cables), or not (I.e. your HDD is truly buggered).

Once you are working on reliable hardware, we can provide further guidance on how to repair your repo.

Damo

3

This looks like the filesystem is damaged. This not a restic issue and there should be no way how restic could cause that. Run fsck (not a restic command) to try to repair the filesystem.

The blame for the i/o error entirely resides with the OS or the underlying hardware.

4

Hello,

After ten days of writing incantations on the terminal in a dimly lit room, I think I’ve managed to isolate the hardware issue, or maybe I’m just going insane, but these are my findings:

The problematic drive, let’s call it alpha, is a 1TB external drive without external power connected through USB. Originally it looked like a hardware issue and then it started looking like a filesystem issue. While I haven’t been able to solve the issue and I think it actually has not solution, I’ve managed to stop the thing that I think caused the issue in the first place.

My first troubleshooting step was through connections, tried 3 different cables and 7 different USB ports (both 2.0 and 3.0) all with the same result, that particular /bf folder that cannot be read.

Next step was trying a different drive, I happen to have another external drive, 2TB, externally powered, which also happens to have a restic repository which comes from the same data as alpha, so with a particular cable and USB port I tested a check → backup → check sequence and everything went perfectly, so cable, port and drive all work (I used same cable port for all of the next tests).

Having determined that the issue is very likely the alpha drive and that the issue was very likely a sector issue I ran many, many tests:

  • smartctl: conveyance, short, long → absolutely no issues, the drive is in great shape, not a single sector unaccounted for
  • badblocks: RO, RW non-destructive, RW destructive (4 passes) → in those 6 passes, not a single error was found, the drive is immaculate

However, the error was still there. Between some of this tests I had to clone the drive to make sure I could get back to the repository to keep troubleshooting and I also had some interesting findings here:

  • rsync was my first attempt at cloning the drive, it was supposed to work, until it couldn’t read the /bf folder, same way you can’t cd into it, so I got a full copy of the repository without that folder in each snapshot which I guess would have broken the repository ever further and wouldn’t let me investigate the issue further
  • ddresuce was my second try, I managed to clone the whole drive sector by sector, I ended up with an identical copy on yet another drive, a 4TB internal drive connected through a very good enclosure that even allows for SMART commands to pass through. Strangely enough, or not, this cloned drive retained the issue, I still could not check the contents of the /bf folder which means the problem is not exactly damaged hardware, but damaged data due to something hardware, which looks a lot like a filesystem issue and that was my next stop

After completely obliterating my drive with the extra strong 4-pass badblocks run, I plugged ddrescue backwards and put all of the sectors back where they were, restoring the original alpha drive, everything worked fine, at least as fine as before badblocks anyway. I proceeded to try fsck, or as NTFS likes to call it ntfsfix, nothing happened, it fixed nothing. I kept asking chatgpt for alternatives and it said that it might be worthwhile to plug the drive into windows and give chkdsk a try which I was already too tired to try so I didn’t go this route although I might need to do it as some point.

This was no the end of it, while running all my tests I had dmesg -wH up to try and catch the hardware errors but I never caught any when trying to read /bf and that bugged me a lot, however, by sheer luck and staring at the screen for a lot of a hours I noticed one weird thing that alpha does when running restic check --read-data and only with --read-data, attached below:

[Oct21 12:14] usb 6-1: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
[  +0.019972] sd 10:0:0:0: [sdh] tag#0 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK cmd_age=0s
[  +0.000006] sd 10:0:0:0: [sdh] tag#0 CDB: Read(10) 28 00 50 1f 77 d8 00 02 00 00
[  +0.000004] blk_update_request: I/O error, dev sdh, sector 1344239576 op 0x0:(READ) flags 0x80700 phys_seg 18 prio class 0
[Oct21 12:25] usb 6-1: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
[  +0.020237] sd 10:0:0:0: [sdh] tag#0 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK cmd_age=0s
[  +0.000016] sd 10:0:0:0: [sdh] tag#0 CDB: Read(10) 28 00 64 3f 4c c0 00 02 00 00
[  +0.000006] blk_update_request: I/O error, dev sdh, sector 1681870016 op 0x0:(READ) flags 0x80700 phys_seg 8 prio class 0
[Oct21 12:26] usb 6-1: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
[  +0.020411] sd 10:0:0:0: [sdh] tag#0 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK cmd_age=0s
[  +0.000027] sd 10:0:0:0: [sdh] tag#0 CDB: Read(10) 28 00 54 02 c3 c0 00 01 00 00
[  +0.000005] blk_update_request: I/O error, dev sdh, sector 1409467328 op 0x0:(READ) flags 0x80700 phys_seg 4 prio class 0
[Oct21 12:41] usb 6-1: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
[  +0.020098] sd 10:0:0:0: [sdh] tag#0 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK cmd_age=0s
[  +0.000006] sd 10:0:0:0: [sdh] tag#0 CDB: Read(10) 28 00 2b ad 14 c0 00 02 00 00
[  +0.000005] blk_update_request: I/O error, dev sdh, sector 732763328 op 0x0:(READ) flags 0x80700 phys_seg 64 prio class 0
[  +1.535653] usb 6-1: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
[  +0.020239] sd 10:0:0:0: [sdh] tag#0 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK cmd_age=0s
[  +0.000005] sd 10:0:0:0: [sdh] tag#0 CDB: Read(10) 28 00 3b 04 ce c0 00 02 00 00
[  +0.000004] blk_update_request: I/O error, dev sdh, sector 990170816 op 0x0:(READ) flags 0x80700 phys_seg 62 prio class 0
[Oct21 12:43] usb 6-1: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
[  +0.019941] sd 10:0:0:0: [sdh] tag#0 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK cmd_age=0s
[  +0.000005] sd 10:0:0:0: [sdh] tag#0 CDB: Read(10) 28 00 17 b7 96 c0 00 02 00 00
[  +0.000004] blk_update_request: I/O error, dev sdh, sector 397907648 op 0x0:(READ) flags 0x80700 phys_seg 64 prio class 0
[Oct21 13:00] usb 6-1: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
[  +0.019970] sd 10:0:0:0: [sdh] tag#0 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK cmd_age=0s
[  +0.000005] sd 10:0:0:0: [sdh] tag#0 CDB: Read(10) 28 00 11 f7 44 e8 00 02 00 00
[  +0.000003] blk_update_request: I/O error, dev sdh, sector 301417704 op 0x0:(READ) flags 0x80700 phys_seg 32 prio class 0

I get a bunch of these resets, followed by a I/O error, it happens on different sectors each time and the sectors are completely readable by dd so I’m very confused by this point, but I’m just guessing that for whatever reason a reset happens, one read fails, probably the operating system takes care of it and re-reads or something like that because this has never caused any issues ever (this drive used to have a complete copy of the original dataset that I would rsync weekly and I did restore from that copy, it worked flawlessly).

I go back to chatgpt in desperation, begging for an answer and the stupid bots gets back to me with the stupidest thing ever that apparently works. It said that sometimes external drives can loose power or not get enough power just from USB itself, even when there’s no way to supply power externally, it just refuses to work sometimes because it’s small.

I just finished a full restic check --read-data through an externally powered USB hub connected to the host, the repository is still in bits and pieces, but I don’t see any more disconnects, no errors, so I’m hoping this counts as stable and reliable hardware and we can find a way to fix the repository.

My professional and completely educated conclusion is that by a mix of: the stupid drive randomly disconnecting, the fact that TIL that you should sync after umount and before disconnecting the drive like a savage, that you can go even further than sync with udsksctl power-off -b /dev/sdX, cosmic rays, rotational velocidensity and sheer bad luck some data got borked.

Can we stabilize the repository enough so I can backup on top of it and maybe just purge/collapse the damaged snapshots or should I wipe the drive and start over? (this time with the stupid and externally powered USB hub).

Let me know, thanks in advance, it’s been fun!

2 Likes
5

Congrats on diagnosing your hardware issue. Consumer hardware is always painful to troubleshoot.

The power issue is likely related to your computer usb hardware. Either you have too many usb devices drawing power; or, your usb hardware (ie your computer) isn’t supplying enough power.

I actually think your Restic repository is probably okay.

Check your volume first. For zfs you need to perform a scrub.

Then run your full Restic check.

Let us know how it goes.

And upgrade your Restic to latest version if you haven’t already. Lots and lots of improvements since 0.11

D

6

Hello,

I automatically check my ZFS pool every two weeks or so, I’ve had data issues with failing drives but not for a long time and certainly not now:

root@lilchewchew2-0:~# zpool status
  pool: gringotts
 state: ONLINE
  scan: scrub repaired 0B in 16:42:38 with 0 errors on Sun Oct 13 17:06:40 2024
config:

	NAME                                 STATE     READ WRITE CKSUM
	gringotts                            ONLINE       0     0     0
	 raidz2-0                           ONLINE       0     0     0
	   ata-ST8000VN004-2M2101_WSD3C3KH  ONLINE       0     0     0
	   ata-ST8000VN004-2M2101_WSD3PWW3  ONLINE       0     0     0
	   ata-ST8000VN004-2M2101_WKD31SR3  ONLINE       0     0     0
	   ata-ST8000NE001-2M7101_WSD42629  ONLINE       0     0     0
	   ata-ST8000VN004-2M2101_WSD46GRJ  ONLINE       0     0     0
	   ata-ST8000VN004-2M2101_WSD1HAVC  ONLINE       0     0     0

errors: No known data errors

I performed a new restic check --read-data with this command

nohup 123restic -r /mnt/alpha/alpha-backup --password-file restic-password.txt check --read-data 2>&1 | tee restic-check-final.log &

just to be sure and I’m still getting the same errors as previously, although this time I get no errors on dmesg so it’s not actually hardware related, this is a summary of the output:

nohup: ignoring input
using temporary cache in /tmp/restic-check-cache-691637299
create exclusive lock for repository
load indexes
check all packs
List(data) returned error, retrying after 1.335327714s: readdirent /mnt/alpha/alpha-backup/data/bf: input/output error
List(data) returned error, retrying after 2.604266718s: readdirent /mnt/alpha/alpha-backup/data/bf: input/output error
List(data) returned error, retrying after 3.822165787s: readdirent /mnt/alpha/alpha-backup/data/bf: input/output error
List(data) returned error, retrying after 9.789308136s: readdirent /mnt/alpha/alpha-backup/data/bf: input/output error
List(data) returned error, retrying after 14.61358065s: readdirent /mnt/alpha/alpha-backup/data/bf: input/output error
List(data) returned error, retrying after 23.12145261s: readdirent /mnt/alpha/alpha-backup/data/bf: input/output error
List(data) returned error, retrying after 1m15.176909413s: readdirent /mnt/alpha/alpha-backup/data/bf: input/output error
List(data) returned error, retrying after 31.681282279s: readdirent /mnt/alpha/alpha-backup/data/bf: input/output error
List(data) returned error, retrying after 32.199183357s: readdirent /mnt/alpha/alpha-backup/data/bf: input/output error
List(data) returned error, retrying after 1m24.82135612s: readdirent /mnt/alpha/alpha-backup/data/bf: input/output error
List(data) returned error, retrying after 1m27.289431553s: readdirent /mnt/alpha/alpha-backup/data/bf: input/output error
List(data) returned error, retrying after 30.707081165s: readdirent /mnt/alpha/alpha-backup/data/bf: input/output error
List(data) returned error, retrying after 38.537129755s: readdirent /mnt/alpha/alpha-backup/data/bf: input/output error
List(data) returned error, retrying after 48.012480824s: readdirent /mnt/alpha/alpha-backup/data/bf: input/output error
List(data) returned error, retrying after 1m19.860007498s: readdirent /mnt/alpha/alpha-backup/data/bf: input/output error
List(data) returned error, retrying after 1m27.433183681s: readdirent /mnt/alpha/alpha-backup/data/bf: input/output error
List(data) returned error, retrying after 1m10.558236298s: readdirent /mnt/alpha/alpha-backup/data/bf: input/output error
List(data) failed: readdirent /mnt/alpha/alpha-backup/data/bf: input/output error
readdirent /mnt/alpha/alpha-backup/data/bf: input/output error
pack ec26d60b3609b71661c7da29090f730aee09b56511a72e06dc01c64136716ee7: does not exist

<repeats a lot for different packs>

pack c67752ced7f4b11ed08f041c31adffb995269ca81c9609a89f975c5b38e2c1f3: does not exist
check snapshots, trees and blobs
[0:11] 100.00%  13 / 13 snapshots
read all data
pack d0b0d0de9042b7be668ef687f46207b8fe917157e839c2d00d7769fd162e1ae0 contains 2 errors: [blob f69c51bbd3adf21663c0c3da7e47ea5b6c1cc3538c14105101c0d67f4ddfaa7c: decrypting blob <data/f69c51bb> from d0b0d0de failed: ciphertext verification failed unexpected pack id 536ca2e9f8b98da0a9900621f6ffc2f8d0037d0a7524ccc33574c20d9a6fdbce]
pack 71923498dfb15926158e06705726517d829a9493a5c5a90e8abbe7c135bd7b31 contains 2 errors: [blob a96d17b7ff2b325f060ec47f3781f5fd2914499632aa540d52c5c8135517532c: decrypting blob <data/a96d17b7> from 71923498 failed: ciphertext verification failed unexpected pack id 3980629925e63046bba1b3a81a7055044d639d5c7f923a4271ee3e20a07dff8c]
pack 3191ef8002a19dc8790e88e5dba8130d3726fe0215cc36f2a33068cc6cb950eb contains 2 errors: [blob 0748a4bf40a1a001cd933e2f8a37b0eb80fef29358b6d4def70d0b0e9b94ab31: decrypting blob <data/0748a4bf> from 3191ef80 failed: ciphertext verification failed unexpected pack id 732ebc641c73a55814110f31f23a86a1aaea417292727eaab101abdc185692fb]
pack 52d8613387a99671f3cb44f04db51914cd5ee100b077065b94c3b587414fe8f1 contains 2 errors: [blob e6fbeff2ac7d2efd7c9976a291fb4ab2d71f71669e0101ee3d530ead85a58613: decrypting blob <data/e6fbeff2> from 52d86133 failed: ciphertext verification failed unexpected pack id ce823dcf61e1841ea576007700dd16e4b74cc39cf0003d16ed00c242fb571d60]
pack 32c369f62640df226429b365d2e25f73f8ad2e22d16de8d4be69be37509fb1c7 contains 2 errors: [blob 3dddb4ca74459ebd3be93bfb3dde2086188d5897e7ff8e8debc527f2a4618a66: decrypting blob <data/3dddb4ca> from 32c369f6 failed: nonce is invalid unexpected pack id 479e32217520e7c7b79a88973d13f92fc2b8b58be21ec580876dd594e397b365]
pack c33c747ab0a73fdf5c79c4a1403d960afedb84b5745e7b0d8fe4658f7223da28 contains 2 errors: [blob 3207e176983edc9af981f551caac72b1d6041b7bc57e96e03e16c528cc891aad: decrypting blob <data/3207e176> from c33c747a failed: ciphertext verification failed unexpected pack id 1781f468366cccb7694ae05436f83f922f981879b3bf93bec1ac595791e9de3b]
pack f3ea14cd7332ecd1c9abb556bb4fab6339ad7213f82fa40240acf081175d4e16 contains 2 errors: [blob 14ec8230727fba63f6c03cbcefdb4b1e99452e203e19edaa01fd7be2518886e9: decrypting blob <data/14ec8230> from f3ea14cd failed: ciphertext verification failed unexpected pack id 3ad8e0c0e591d21423a0ab0c927686ffcb931753c5dbfb9ba15f8fc79b4a61a0]
pack 43c19e0d42730145e5463d643b1fc16ce175004c4c12f2913dd3a28c9cbe90dd contains 2 errors: [blob 0ce72796d4c99b52f7f2a03f0ee86a01a32d4d563c0c778229fab63836d3a6c3: decrypting blob <data/0ce72796> from 43c19e0d failed: ciphertext verification failed unexpected pack id e610da91bbf0115867813b0a14b540316dec7a5096bd0935682e0ac33b23126c]
pack a3c5824930c11f4ade412a49995d465cf434e1379d0a0a7adfb4e8ef1ef6b886 contains 2 errors: [blob bc69b0dab67e890a59d3376a870a950ac94a5e27cdc3d22c982c480a165b2bf9: decrypting blob <data/bc69b0da> from a3c58249 failed: ciphertext verification failed unexpected pack id 2c40edb626fc70766521b4db42ddfe90bf6ea43939ca85d0b293dfeb5b51fb2c]
pack 34b15eb1dc61fea1b674f93f1b081ac6de94ae67c365d0f9534d66e3b4c7a1a1 contains 2 errors: [blob 67cb7560417a1e57fd2435cc65c25c9374dff0dc490afbf128eb4abf2dd72e27: decrypting blob <data/67cb7560> from 34b15eb1 failed: ciphertext verification failed unexpected pack id 2fa2f4c6f6ef8a680cddb2c0a351ea05de2546def6244feb81a8187c9ece7894]
pack 15aeaf95d3ded827ae1d5b55b21e6363cad6d67cc81c70e9a14ca0b2c950729d contains 2 errors: [blob b9546de7716b5880ba06c02967ef52ed2fa3b81aebc22f413051f126b8583197: decrypting blob <data/b9546de7> from 15aeaf95 failed: ciphertext verification failed unexpected pack id dfd65f3089f4a9483f0cf04475b094f0f0cd6661ab6704654eeb9c34187e6375]
pack c5037995c681f437d311a89e6e3ab9f3115e395953f631d1f9038079284a4c5c contains 2 errors: [blob b171853902d4a98c97eb60f8db96d544172bc76409bc16b1a988c53f537185e9: decrypting blob <data/b1718539> from c5037995 failed: ciphertext verification failed unexpected pack id 1d678e94b1671c47003bce489750739da111dbb6e0af32933a1faf8ac36197aa]
pack 05df8e387f2804f7f4e7ba28e6c6ab8408b2a51ac34c6347d0e79fc5867b6623 contains 2 errors: [blob 86e8679f7129e1e3b96fcc74a13b0eef7adb501422ece138fde52883cc74ee23: decrypting blob <data/86e8679f> from 05df8e38 failed: ciphertext verification failed unexpected pack id 105a1f0546213701482a102b460ef98a1ce5ca9950c92448b8dbc223d855ac92]
pack 15ed3deb342d5ab683fea7577b49b76e5ed1dc1098955755af8ebb59831f8b53 contains 2 errors: [blob c2d670c596c78443875529fd29912980470c20f99f224b55d6e5cad455b2c8a7: decrypting blob <data/c2d670c5> from 15ed3deb failed: ciphertext verification failed unexpected pack id 33998ad7f75b862366c5fb4687fd8803ecdfd1ddd7a9485d6dab6f5fd6fff9d5]
pack 16a81b3e7affb66ed02490eec34386ab087a01b593ece61922200fd55eba4297 contains 2 errors: [blob 5899f49acd08c4c126905adc8885702e66522342b709047bac300d8bede83b26: decrypting blob <data/5899f49a> from 16a81b3e failed: ciphertext verification failed unexpected pack id 7a3a5c3676797795b7e9365c4512f3b9a9eb3f1625ba637749733938d0178a31]
pack 161484e63a65d345681ce7e9334c63289928381250bf45c7956aa741315d9f58 contains 2 errors: [blob da0c4312d8d2d4ffbd8b97fec609ae5ece29889e2c1a276e65de686f3befaf16: decrypting blob <data/da0c4312> from 161484e6 failed: ciphertext verification failed unexpected pack id e89b7daf827e561960a1892a9b8cdd9e71fb8443520d3515e59f6c78758b9c65]
pack 067803e995685a4b1516add892c7dad7a00e6fc15c3323ae79c94f809a357297 contains 2 errors: [blob c5d9059f69a622875850adcd02aa53ca37643384cb4593ffc09cb8d21feda6d2: decrypting blob <data/c5d9059f> from 067803e9 failed: ciphertext verification failed unexpected pack id cf0d8840d9d962c1f793a6aace086a6be5a40ebb24330a1f05726eded94955ce]
pack c6cd7efe7193b6ef41d7007a34f12262b6f9f9f3f34b8ecd541e5a8995fe1eb5 contains 2 errors: [blob 331a44e3a225c51440467c967cf0fecea898724d86d2ed4b56e22b46cb72d019: decrypting blob <data/331a44e3> from c6cd7efe failed: ciphertext verification failed unexpected pack id 41246b97f0f3f3d80afdcbde3b0ba7c0b6c6dafc9615de463c85421dc7be05fc]
pack 679fccd2fd59e73bc13f7c6f2aab2ec3ad2b8748c19d9a39cd93c7735cef4a1e contains 2 errors: [blob 63f384b182ee0946b3a0729d58d2ae1d917e36d25b2945b6a3703ae8b975a6c6: decrypting blob <data/63f384b1> from 679fccd2 failed: ciphertext verification failed unexpected pack id e40a32b8e69fa8f81b13e121c67dec8d8a10306536694d87f205c89df7f99088]
pack b88225fae4749651fb7eae99a5c5477f178c74e9a0187f498a58fcad1084a4a9 contains 2 errors: [blob c722e8bc16acc10d66e797b57a21162bf2fba93ab573e1bf9dfd82cf600fb57f: decrypting blob <data/c722e8bc> from b88225fa failed: ciphertext verification failed unexpected pack id 93a9352e1179c3fbe633bc8dcdf2efee548b7c391515c58ed6e1cc4517fd0f78]
pack 0837357f89698c36a3e78279d7ef212d9151cd4a91338705f4583c53f1f10ad9 contains 2 errors: [blob 1f3bc4912ed7962c94dfa7e9d38609bfda32e533be9f4ddacada477600bce0cb: decrypting blob <data/1f3bc491> from 0837357f failed: ciphertext verification failed unexpected pack id b2cefe8ac14b653e4a4ff59e3ef243c5f9e2a50e6ad5b8e2524dc39662191d54]
pack b8cf64cd61e59a9fd2fd12eea45e3765dc7f519a27d3eac2bebbeaaac601b704 contains 2 errors: [blob d2b11592ed3d772d7b9f8e21407878e961bada4bc34154684e69728f5f69270e: decrypting blob <data/d2b11592> from b8cf64cd failed: ciphertext verification failed unexpected pack id aeaa8f5c1e35e72b33edf41f116fa1c03a1c2d15c22e6ba0e1a692807f334018]
pack c90da86133c5e3678cefbd50a4ffa14d92a1a3d3e1821ad89e9d95d1a2790fbc contains 2 errors: [blob 1f0d6926430600a3cd2ae6f9e669650ac3ba5879e701de0cac9cf9b220938ade: decrypting blob <data/1f0d6926> from c90da861 failed: ciphertext verification failed unexpected pack id 665fd1cf3066fe230c14b0eb1e6fd3c57a2d5182662dad152cab06e01cd1db8a]
pack 5b5e0692d2d828caa35c84d1d7fdd7743afee69a5eeb97fc03b49c76eb10abef contains 2 errors: [blob 6aaf40065d49c857cfa725df59de058d38924449f0dcb311ed0682741a39f649: decrypting blob <data/6aaf4006> from 5b5e0692 failed: ciphertext verification failed unexpected pack id ec8537b9cd67a0ff55e88240f3bdf48fba80e633994f2334fc5ebfe701ed4a8b]
pack 1b07b57b52391e630792aaf85dc13dc6086bc8f776c22f7b5e4a86c7c5d73aea contains 2 errors: [blob aafbb6e7299ac6e69da06fef47f9cfe2dacbb1f7dfd8c95a357bdd10770560f3: decrypting blob <data/aafbb6e7> from 1b07b57b failed: ciphertext verification failed unexpected pack id f880dbce0e14fd4a10f3d13a956ca61257337cefddd4b017f55a2c62a2429f0a]
pack 1cc6e43b5ba4050ed723e5319533e9b1fa5349b1f715c8e00fa080ea54e2fbb7 contains 2 errors: [blob 4904a9747948d360a6454ecf7f349c8f248a67ea9d6b6e7fcdac26ff8ef4f9d0: decrypting blob <data/4904a974> from 1cc6e43b failed: ciphertext verification failed unexpected pack id 850beb18d0dc72e371bde20c28b40ce664a3ec17663f0bc02af37b12870a84ed]
pack 8c00fe576a96b4bbce3deb4a0f755e4fc32af242252175f664c20cfc8bb35238 contains 2 errors: [blob 28afe7eb6966730936558a742649de2151ba86b00eeaca2c3173657c7b287803: decrypting blob <data/28afe7eb> from 8c00fe57 failed: ciphertext verification failed unexpected pack id 02a09df7cd808eaa8c75d81bee41c8df0261ddf2322ad204b0e2b57b07408889]
pack 7ccfa4fbc681a1f81e25706e2fc01b7fa24679532ba42ed5e769b07af2532703 contains 2 errors: [blob 7299f09abe66f430cd98380177c4b8bf00269a15629cd0645834d26b4c927103: decrypting blob <data/7299f09a> from 7ccfa4fb failed: ciphertext verification failed unexpected pack id 5fafd762e4f302b0c8147861367915f15fcda5cad2472ab7a1ebc283b258b59e]
pack 4c5f4c9146f07860d61d99f5749cb3f3e1fe5f3e8de704c30c90fe76a6e46f49 contains 2 errors: [blob 82cff50ff9990efcc0900802403ecb62d8f4669a27081ebbb0afeac4b4ac7b1f: decrypting blob <data/82cff50f> from 4c5f4c91 failed: ciphertext verification failed unexpected pack id 1ed0ed686aaa9cba2f2c7463cc5bd5855c6b5aed1b1caf825d6df2ac9c7240e4]
pack 5c1d153cc57b04574538788fd13dcee7caca382c2888d4084c0ec7155d25c011 contains 2 errors: [blob 095616b34c05c7c203953f302dfa1473f78f74de4cc2346c074f6b311d39d964: decrypting blob <data/095616b3> from 5c1d153c failed: ciphertext verification failed unexpected pack id ca7088c36b8c5183da1dbae4a16bc83a687faf868b0116c68c6a23140b307eaa]
pack 4d0bee84353bf456b7455839c9cb25e667c71e367461393c59848a27dcdb5abc contains 2 errors: [blob 0b74ccc1c27781343aaf606700bf4dbbe1e4cc63fe5768ba27647903a1bf7017: decrypting blob <data/0b74ccc1> from 4d0bee84 failed: ciphertext verification failed unexpected pack id 50731774078d2089e420854195f997ff2d361a86fb04ce7610905e8fbeb90929]
pack ceb51a9c1a9fa14c3f7fa91593bb7d0c07adc5bdb0c43f9701560df63070cb8b contains 2 errors: [blob ce0473cfa22230e996ff4bee5fb7ed2ee3c84e9804a07a2176be5b73ff421acc: decrypting blob <data/ce0473cf> from ceb51a9c failed: ciphertext verification failed unexpected pack id 5e28533102daa404029f905d07a6ea73b41974c8e0cc57b09fba39fdfb831fcf]
pack 7e3b98ac39c7b6aafd9c794c7460d72645138576723d622456e9b6fcfdb3f83e contains 2 errors: [blob 2d309f656ecf7c03c03e64f5b7da2b9194668cef8bc715b43f3f8fb3c42d60c1: decrypting blob <data/2d309f65> from 7e3b98ac failed: ciphertext verification failed unexpected pack id cd972f41dc26c70c3fb3a748c6f86ec5cd5b3345384664cb149f51bc06e6d5b3]
pack 8eb64ec7051ca0e0bba85aad27989638881997f73cd04fd67da1f705abe4a8e8 contains 2 errors: [blob 6ab7ef8f39598599f36ddfa54d765fc1697a3fafb824319d2de5fd903b05cf03: decrypting blob <data/6ab7ef8f> from 8eb64ec7 failed: ciphertext verification failed unexpected pack id d76ab7d5c79f71f41fa05cbba401729286b016a403e9afbbf9e92898411c49df]
Load(<data/bf21c3286e>, 4546745, 0) failed: open /mnt/alpha/alpha-backup/data/bf/bf21c3286ef353b85de53f2d74009b1b707dee26bb188c82171dac7a94bba18f: no such file or directory
Load(<data/bf21c3286e>, 4546745, 0) failed: open /mnt/alpha/alpha-backup/data/bf/bf21c3286ef353b85de53f2d74009b1b707dee26bb188c82171dac7a94bba18f: no such file or directory
download error: open /mnt/alpha/alpha-backup/data/bf/bf21c3286ef353b85de53f2d74009b1b707dee26bb188c82171dac7a94bba18f: no such file or directory
Load(<data/bf1cadf62e>, 4729363, 0) failed: open /mnt/alpha/alpha-backup/data/bf/bf1cadf62e0372f741dfbcf431953106c7ef217669e28f202a42423708c62076: no such file or directory
Load(<data/bf1cadf62e>, 4729363, 0) failed: open /mnt/alpha/alpha-backup/data/bf/bf1cadf62e0372f741dfbcf431953106c7ef217669e28f202a42423708c62076: no such file or directory
download error: open /mnt/alpha/alpha-backup/data/bf/bf1cadf62e0372f741dfbcf431953106c7ef217669e28f202a42423708c62076: no such file or directory
Load(<data/bf1ebf05d4>, 4754151, 0) failed: open /mnt/alpha/alpha-backup/data/bf/bf1ebf05d4871d9dcdd02a47f74535e680cb42d511368d1baf6e71c380f94712: no such file or directory
Load(<data/bf1ebf05d4>, 4754151, 0) failed: open /mnt/alpha/alpha-backup/data/bf/bf1ebf05d4871d9dcdd02a47f74535e680cb42d511368d1baf6e71c380f94712: no such file or directory
download error: open /mnt/alpha/alpha-backup/data/bf/bf1ebf05d4871d9dcdd02a47f74535e680cb42d511368d1baf6e71c380f94712: no such file or directory
pack bf14413f9d25af261ac92504e04e2b86e97ebdd4801abf15d7d0beeee1c56854 contains 2 errors: [blob f83c0a424bd57cc2233743131a8346a5b1a2878a5ed02250ca05376cdd34b8fa: decrypting blob <data/f83c0a42> from bf14413f failed: nonce is invalid unexpected pack id 565befe7ac4c9bc46ecb830e1fcde00aad7e20db8f998a83cb83f1fd42cfafae]
Load(<data/bf21e5ae0e>, 5297179, 0) failed: open /mnt/alpha/alpha-backup/data/bf/bf21e5ae0e0713ad7b457c2e3dac04fd8f2efdd6a8475836bed3ca2d0af45146: no such file or directory
Load(<data/bf21e5ae0e>, 5297179, 0) failed: open /mnt/alpha/alpha-backup/data/bf/bf21e5ae0e0713ad7b457c2e3dac04fd8f2efdd6a8475836bed3ca2d0af45146: no such file or directory
download error: open /mnt/alpha/alpha-backup/data/bf/bf21e5ae0e0713ad7b457c2e3dac04fd8f2efdd6a8475836bed3ca2d0af45146: no such file or directory
Load(<data/bf1fc38c63>, 4808236, 0) failed: open /mnt/alpha/alpha-backup/data/bf/bf1fc38c63ee13de90ef9cf573331ee90c21296c3351b19b24c1373e51b5b766: no such file or directory
Load(<data/bf1fc38c63>, 4808236, 0) failed: open /mnt/alpha/alpha-backup/data/bf/bf1fc38c63ee13de90ef9cf573331ee90c21296c3351b19b24c1373e51b5b766: no such file or directory
download error: open /mnt/alpha/alpha-backup/data/bf/bf1fc38c63ee13de90ef9cf573331ee90c21296c3351b19b24c1373e51b5b766: no such file or directory
Load(<data/bf229d9872>, 4801376, 0) failed: open /mnt/alpha/alpha-backup/data/bf/bf229d98728e10ab6cbccc81c4d8133cbf8f64a2a22cf01f65c340f8c445fdab: no such file or directory
Load(<data/bf229d9872>, 4801376, 0) failed: open /mnt/alpha/alpha-backup/data/bf/bf229d98728e10ab6cbccc81c4d8133cbf8f64a2a22cf01f65c340f8c445fdab: no such file or directory
download error: open /mnt/alpha/alpha-backup/data/bf/bf229d98728e10ab6cbccc81c4d8133cbf8f64a2a22cf01f65c340f8c445fdab: no such file or directory
pack afcd2309393dc03c4e5bff6539d85947aa6935a2d13a5d2fca78145e800f90fb contains 2 errors: [blob 0d49f654b435fd82faa67d3399a365c7bec27165afa0d3ca11fd46d0b62f8322: decrypting blob <data/0d49f654> from afcd2309 failed: ciphertext verification failed unexpected pack id c993346a1aa636d9da61f4a3cb6f1458c03c23caead5b01b68a129a6e4020a7a]
Load(<data/bf2028c95f>, 4450257, 0) failed: open /mnt/alpha/alpha-backup/data/bf/bf2028c95f536475fa3d4168258cf307e8c1b2e413ca00d57c9146862c802b71: no such file or directory
Load(<data/bf2028c95f>, 4450257, 0) failed: open /mnt/alpha/alpha-backup/data/bf/bf2028c95f536475fa3d4168258cf307e8c1b2e413ca00d57c9146862c802b71: no such file or directory
download error: open /mnt/alpha/alpha-backup/data/bf/bf2028c95f536475fa3d4168258cf307e8c1b2e413ca00d57c9146862c802b71: no such file or directory
Load(<data/bf1c27335e>, 4729100, 0) failed: open /mnt/alpha/alpha-backup/data/bf/bf1c27335e6b19f33edb2d1bef36268ada2e1f437de5867a77b3cc430ea9a941: no such file or directory
Load(<data/bf1c27335e>, 4729100, 0) failed: open /mnt/alpha/alpha-backup/data/bf/bf1c27335e6b19f33edb2d1bef36268ada2e1f437de5867a77b3cc430ea9a941: no such file or directory
download error: open /mnt/alpha/alpha-backup/data/bf/bf1c27335e6b19f33edb2d1bef36268ada2e1f437de5867a77b3cc430ea9a941: no such file or directory
[6:19:48] 100.00%  81729 / 81729 packs

The repository contains damaged pack files. These damaged files must be removed to repair the repository. This can be done using the following commands. Please read the troubleshooting guide at https://restic.readthedocs.io/en/stable/077_troubleshooting.html first.

restic repair packs c5037995c681f437d311a89e6e3ab9f3115e395953f631d1f9038079284a4c5c bf14413f9d25af261ac92504e04e2b86e97ebdd4801abf15d7d0beeee1c56854 52d8613387a99671f3cb44f04db51914cd5ee100b077065b94c3b587414fe8f1 161484e63a65d345681ce7e9334c63289928381250bf45c7956aa741315d9f58 0837357f89698c36a3e78279d7ef212d9151cd4a91338705f4583c53f1f10ad9 4c5f4c9146f07860d61d99f5749cb3f3e1fe5f3e8de704c30c90fe76a6e46f49 34b15eb1dc61fea1b674f93f1b081ac6de94ae67c365d0f9534d66e3b4c7a1a1 15ed3deb342d5ab683fea7577b49b76e5ed1dc1098955755af8ebb59831f8b53 32c369f62640df226429b365d2e25f73f8ad2e22d16de8d4be69be37509fb1c7 a3c5824930c11f4ade412a49995d465cf434e1379d0a0a7adfb4e8ef1ef6b886 16a81b3e7affb66ed02490eec34386ab087a01b593ece61922200fd55eba4297 679fccd2fd59e73bc13f7c6f2aab2ec3ad2b8748c19d9a39cd93c7735cef4a1e c90da86133c5e3678cefbd50a4ffa14d92a1a3d3e1821ad89e9d95d1a2790fbc 1cc6e43b5ba4050ed723e5319533e9b1fa5349b1f715c8e00fa080ea54e2fbb7 8c00fe576a96b4bbce3deb4a0f755e4fc32af242252175f664c20cfc8bb35238 3191ef8002a19dc8790e88e5dba8130d3726fe0215cc36f2a33068cc6cb950eb ceb51a9c1a9fa14c3f7fa91593bb7d0c07adc5bdb0c43f9701560df63070cb8b afcd2309393dc03c4e5bff6539d85947aa6935a2d13a5d2fca78145e800f90fb 5c1d153cc57b04574538788fd13dcee7caca382c2888d4084c0ec7155d25c011 c33c747ab0a73fdf5c79c4a1403d960afedb84b5745e7b0d8fe4658f7223da28 05df8e387f2804f7f4e7ba28e6c6ab8408b2a51ac34c6347d0e79fc5867b6623 b88225fae4749651fb7eae99a5c5477f178c74e9a0187f498a58fcad1084a4a9 5b5e0692d2d828caa35c84d1d7fdd7743afee69a5eeb97fc03b49c76eb10abef 4d0bee84353bf456b7455839c9cb25e667c71e367461393c59848a27dcdb5abc 8eb64ec7051ca0e0bba85aad27989638881997f73cd04fd67da1f705abe4a8e8 d0b0d0de9042b7be668ef687f46207b8fe917157e839c2d00d7769fd162e1ae0 067803e995685a4b1516add892c7dad7a00e6fc15c3323ae79c94f809a357297 7ccfa4fbc681a1f81e25706e2fc01b7fa24679532ba42ed5e769b07af2532703 f3ea14cd7332ecd1c9abb556bb4fab6339ad7213f82fa40240acf081175d4e16 15aeaf95d3ded827ae1d5b55b21e6363cad6d67cc81c70e9a14ca0b2c950729d c6cd7efe7193b6ef41d7007a34f12262b6f9f9f3f34b8ecd541e5a8995fe1eb5 1b07b57b52391e630792aaf85dc13dc6086bc8f776c22f7b5e4a86c7c5d73aea 7e3b98ac39c7b6aafd9c794c7460d72645138576723d622456e9b6fcfdb3f83e 71923498dfb15926158e06705726517d829a9493a5c5a90e8abbe7c135bd7b31 b8cf64cd61e59a9fd2fd12eea45e3765dc7f519a27d3eac2bebbeaaac601b704 43c19e0d42730145e5463d643b1fc16ce175004c4c12f2913dd3a28c9cbe90dd
restic repair snapshots --forget

Damaged pack files can be caused by backend problems, hardware problems or bugs in restic. Please open an issue at https://github.com/restic/restic/issues/new/choose for further troubleshooting!
Fatal: repository contains errors

I’ve already updated my restic version I just need to delete the distro provided one and rename my 123restic.

So should I follow the instructions and execute a repair packs... followed by a repair snapshots --forget and try to backup again?

Looking forward to your answer, TIA!

7

Okay, so I have re-read your posts. The zfs pool is your source data, and you are backing up to a restic repository on the external USB 1TB drive, which is mounted at /mnt/alpha/alpha-backup.

Michael is right: you need to run fsck on the drive. But you will probably need to force it to do a full check, rather than just check the journal if your drive is using ext4.

So, do the following:

mount|grep /mnt/alpha

Note which device file it is mounted on… i.e. /dev/sda1

Then

sudo umount /mnt/alpha
sudo fsck -f <device file for your drive>

I expect you will see many errors. Let fsck fix them, so your drive’s metadata is sane.

Then mount the drive again, and run your restic check again.

8

Depending on which ntfs implementation you use, the disconnects could be enough to massively corrupt the filesystem. Restic itself explicitly asks the OS to definitely really write the files to disk before proceeding. That is, if the filesystem honors that request, then a randomly disconnecting drive wouldn’t be a problem. However, from past experience ntfs on Linux seems to be rather brittle in that regard. Please just use a standard filesystem like ext4. (no you don’t need a filesystem that checksums its data. restic flushes the files to disk in the right order that ext4 is enough)

None of this should be necessary, if the filesystem writes the file to disk as it was told to by restic.

There’s not point in doing that while the filesystem is in this broken state. If you have a Windows installation at hand, you could try whether chkdsk has more luck fixing the filesystem.

Otherwise, if all files in the bf folder are missing, this means that each snapshot will miss a significant chunk of data. If you really want to keep the remainders, then copy everything except the bf folder to a different drive and reformat the alpha drive. The filesystem is likely broken to a point where you can no longer rely on it to not loose files.

9

Hello,

Thanks you Damo and Michael. Here’s a progress update.

I did have a w11 machine that I could use and I ran chkdsk there, I kept an eye on the output and noticed that it repaired the index, I’m sure there were other things that were repaired but I left it overnight and windows updated and restarted itself so I can’t tell for sure what happened. I ran it again with logs and it just said it had nothing to repair. I did notice that when I plugged the drive the first time I got a message that it looked broken, after the first run, the message stopped popping up.

I plugged the drive to the NAS and instantly tried to check the /bf folder and it’s there and I can see the files inside. I checked the drive with the usual (making sure to plug it to the externally powered USB hub):

nohup 123restic -r /mnt/alpha/alpha-backup --password-file restic-password.txt check --read-data 2>&1 | tee restic-check-after-chkdsk.log &

And only got a handful of errors from restic, no disconnects or other problems in dmesg:

nohup: ignoring input
using temporary cache in /tmp/restic-check-cache-3645431041
create exclusive lock for repository
load indexes
check all packs
1219 additional files were found in the repo, which likely contain duplicate data.
This is non-critical, you can run `restic prune` to correct this.
check snapshots, trees and blobs
[0:11] 100.00%  13 / 13 snapshots
read all data
pack d0b0d0de9042b7be668ef687f46207b8fe917157e839c2d00d7769fd162e1ae0 contains 2 errors: [blob f69c51bbd3adf21663c0c3da7e47ea5b6c1cc3538c14105101c0d67f4ddfaa7c: decrypting blob <data/f69c51bb> from d0b0d0de failed: ciphertext verification failed unexpected pack id 536ca2e9f8b98da0a9900621f6ffc2f8d0037d0a7524ccc33574c20d9a6fdbce]
pack 3191ef8002a19dc8790e88e5dba8130d3726fe0215cc36f2a33068cc6cb950eb contains 2 errors: [blob 0748a4bf40a1a001cd933e2f8a37b0eb80fef29358b6d4def70d0b0e9b94ab31: decrypting blob <data/0748a4bf> from 3191ef80 failed: ciphertext verification failed unexpected pack id 732ebc641c73a55814110f31f23a86a1aaea417292727eaab101abdc185692fb]
pack 71923498dfb15926158e06705726517d829a9493a5c5a90e8abbe7c135bd7b31 contains 2 errors: [blob a96d17b7ff2b325f060ec47f3781f5fd2914499632aa540d52c5c8135517532c: decrypting blob <data/a96d17b7> from 71923498 failed: ciphertext verification failed unexpected pack id 3980629925e63046bba1b3a81a7055044d639d5c7f923a4271ee3e20a07dff8c]
pack 32c369f62640df226429b365d2e25f73f8ad2e22d16de8d4be69be37509fb1c7 contains 2 errors: [blob 3dddb4ca74459ebd3be93bfb3dde2086188d5897e7ff8e8debc527f2a4618a66: decrypting blob <data/3dddb4ca> from 32c369f6 failed: nonce is invalid unexpected pack id 479e32217520e7c7b79a88973d13f92fc2b8b58be21ec580876dd594e397b365]
pack 52d8613387a99671f3cb44f04db51914cd5ee100b077065b94c3b587414fe8f1 contains 2 errors: [blob e6fbeff2ac7d2efd7c9976a291fb4ab2d71f71669e0101ee3d530ead85a58613: decrypting blob <data/e6fbeff2> from 52d86133 failed: ciphertext verification failed unexpected pack id ce823dcf61e1841ea576007700dd16e4b74cc39cf0003d16ed00c242fb571d60]
pack 43c19e0d42730145e5463d643b1fc16ce175004c4c12f2913dd3a28c9cbe90dd contains 2 errors: [blob 0ce72796d4c99b52f7f2a03f0ee86a01a32d4d563c0c778229fab63836d3a6c3: decrypting blob <data/0ce72796> from 43c19e0d failed: ciphertext verification failed unexpected pack id e610da91bbf0115867813b0a14b540316dec7a5096bd0935682e0ac33b23126c]
pack c33c747ab0a73fdf5c79c4a1403d960afedb84b5745e7b0d8fe4658f7223da28 contains 2 errors: [blob 3207e176983edc9af981f551caac72b1d6041b7bc57e96e03e16c528cc891aad: decrypting blob <data/3207e176> from c33c747a failed: ciphertext verification failed unexpected pack id 1781f468366cccb7694ae05436f83f922f981879b3bf93bec1ac595791e9de3b]
pack a3c5824930c11f4ade412a49995d465cf434e1379d0a0a7adfb4e8ef1ef6b886 contains 2 errors: [blob bc69b0dab67e890a59d3376a870a950ac94a5e27cdc3d22c982c480a165b2bf9: decrypting blob <data/bc69b0da> from a3c58249 failed: ciphertext verification failed unexpected pack id 2c40edb626fc70766521b4db42ddfe90bf6ea43939ca85d0b293dfeb5b51fb2c]
pack f3ea14cd7332ecd1c9abb556bb4fab6339ad7213f82fa40240acf081175d4e16 contains 2 errors: [blob 14ec8230727fba63f6c03cbcefdb4b1e99452e203e19edaa01fd7be2518886e9: decrypting blob <data/14ec8230> from f3ea14cd failed: ciphertext verification failed unexpected pack id 3ad8e0c0e591d21423a0ab0c927686ffcb931753c5dbfb9ba15f8fc79b4a61a0]
pack 34b15eb1dc61fea1b674f93f1b081ac6de94ae67c365d0f9534d66e3b4c7a1a1 contains 2 errors: [blob 67cb7560417a1e57fd2435cc65c25c9374dff0dc490afbf128eb4abf2dd72e27: decrypting blob <data/67cb7560> from 34b15eb1 failed: ciphertext verification failed unexpected pack id 2fa2f4c6f6ef8a680cddb2c0a351ea05de2546def6244feb81a8187c9ece7894]
pack 15ed3deb342d5ab683fea7577b49b76e5ed1dc1098955755af8ebb59831f8b53 contains 2 errors: [blob c2d670c596c78443875529fd29912980470c20f99f224b55d6e5cad455b2c8a7: decrypting blob <data/c2d670c5> from 15ed3deb failed: ciphertext verification failed unexpected pack id 33998ad7f75b862366c5fb4687fd8803ecdfd1ddd7a9485d6dab6f5fd6fff9d5]
pack 15aeaf95d3ded827ae1d5b55b21e6363cad6d67cc81c70e9a14ca0b2c950729d contains 2 errors: [blob b9546de7716b5880ba06c02967ef52ed2fa3b81aebc22f413051f126b8583197: decrypting blob <data/b9546de7> from 15aeaf95 failed: ciphertext verification failed unexpected pack id dfd65f3089f4a9483f0cf04475b094f0f0cd6661ab6704654eeb9c34187e6375]
pack 05df8e387f2804f7f4e7ba28e6c6ab8408b2a51ac34c6347d0e79fc5867b6623 contains 2 errors: [blob 86e8679f7129e1e3b96fcc74a13b0eef7adb501422ece138fde52883cc74ee23: decrypting blob <data/86e8679f> from 05df8e38 failed: ciphertext verification failed unexpected pack id 105a1f0546213701482a102b460ef98a1ce5ca9950c92448b8dbc223d855ac92]
pack c5037995c681f437d311a89e6e3ab9f3115e395953f631d1f9038079284a4c5c contains 2 errors: [blob b171853902d4a98c97eb60f8db96d544172bc76409bc16b1a988c53f537185e9: decrypting blob <data/b1718539> from c5037995 failed: ciphertext verification failed unexpected pack id 1d678e94b1671c47003bce489750739da111dbb6e0af32933a1faf8ac36197aa]
pack 161484e63a65d345681ce7e9334c63289928381250bf45c7956aa741315d9f58 contains 2 errors: [blob da0c4312d8d2d4ffbd8b97fec609ae5ece29889e2c1a276e65de686f3befaf16: decrypting blob <data/da0c4312> from 161484e6 failed: ciphertext verification failed unexpected pack id e89b7daf827e561960a1892a9b8cdd9e71fb8443520d3515e59f6c78758b9c65]
pack 067803e995685a4b1516add892c7dad7a00e6fc15c3323ae79c94f809a357297 contains 2 errors: [blob c5d9059f69a622875850adcd02aa53ca37643384cb4593ffc09cb8d21feda6d2: decrypting blob <data/c5d9059f> from 067803e9 failed: ciphertext verification failed unexpected pack id cf0d8840d9d962c1f793a6aace086a6be5a40ebb24330a1f05726eded94955ce]
pack c6cd7efe7193b6ef41d7007a34f12262b6f9f9f3f34b8ecd541e5a8995fe1eb5 contains 2 errors: [blob 331a44e3a225c51440467c967cf0fecea898724d86d2ed4b56e22b46cb72d019: decrypting blob <data/331a44e3> from c6cd7efe failed: ciphertext verification failed unexpected pack id 41246b97f0f3f3d80afdcbde3b0ba7c0b6c6dafc9615de463c85421dc7be05fc]
pack 16a81b3e7affb66ed02490eec34386ab087a01b593ece61922200fd55eba4297 contains 2 errors: [blob 5899f49acd08c4c126905adc8885702e66522342b709047bac300d8bede83b26: decrypting blob <data/5899f49a> from 16a81b3e failed: ciphertext verification failed unexpected pack id 7a3a5c3676797795b7e9365c4512f3b9a9eb3f1625ba637749733938d0178a31]
pack 679fccd2fd59e73bc13f7c6f2aab2ec3ad2b8748c19d9a39cd93c7735cef4a1e contains 2 errors: [blob 63f384b182ee0946b3a0729d58d2ae1d917e36d25b2945b6a3703ae8b975a6c6: decrypting blob <data/63f384b1> from 679fccd2 failed: ciphertext verification failed unexpected pack id e40a32b8e69fa8f81b13e121c67dec8d8a10306536694d87f205c89df7f99088]
pack b8cf64cd61e59a9fd2fd12eea45e3765dc7f519a27d3eac2bebbeaaac601b704 contains 2 errors: [blob d2b11592ed3d772d7b9f8e21407878e961bada4bc34154684e69728f5f69270e: decrypting blob <data/d2b11592> from b8cf64cd failed: ciphertext verification failed unexpected pack id aeaa8f5c1e35e72b33edf41f116fa1c03a1c2d15c22e6ba0e1a692807f334018]
pack b88225fae4749651fb7eae99a5c5477f178c74e9a0187f498a58fcad1084a4a9 contains 2 errors: [blob c722e8bc16acc10d66e797b57a21162bf2fba93ab573e1bf9dfd82cf600fb57f: decrypting blob <data/c722e8bc> from b88225fa failed: ciphertext verification failed unexpected pack id 93a9352e1179c3fbe633bc8dcdf2efee548b7c391515c58ed6e1cc4517fd0f78]
pack 0837357f89698c36a3e78279d7ef212d9151cd4a91338705f4583c53f1f10ad9 contains 2 errors: [blob 1f3bc4912ed7962c94dfa7e9d38609bfda32e533be9f4ddacada477600bce0cb: decrypting blob <data/1f3bc491> from 0837357f failed: ciphertext verification failed unexpected pack id b2cefe8ac14b653e4a4ff59e3ef243c5f9e2a50e6ad5b8e2524dc39662191d54]
pack c90da86133c5e3678cefbd50a4ffa14d92a1a3d3e1821ad89e9d95d1a2790fbc contains 2 errors: [blob 1f0d6926430600a3cd2ae6f9e669650ac3ba5879e701de0cac9cf9b220938ade: decrypting blob <data/1f0d6926> from c90da861 failed: ciphertext verification failed unexpected pack id 665fd1cf3066fe230c14b0eb1e6fd3c57a2d5182662dad152cab06e01cd1db8a]
pack 1b07b57b52391e630792aaf85dc13dc6086bc8f776c22f7b5e4a86c7c5d73aea contains 2 errors: [blob aafbb6e7299ac6e69da06fef47f9cfe2dacbb1f7dfd8c95a357bdd10770560f3: decrypting blob <data/aafbb6e7> from 1b07b57b failed: ciphertext verification failed unexpected pack id f880dbce0e14fd4a10f3d13a956ca61257337cefddd4b017f55a2c62a2429f0a]
pack 5b5e0692d2d828caa35c84d1d7fdd7743afee69a5eeb97fc03b49c76eb10abef contains 2 errors: [blob 6aaf40065d49c857cfa725df59de058d38924449f0dcb311ed0682741a39f649: decrypting blob <data/6aaf4006> from 5b5e0692 failed: ciphertext verification failed unexpected pack id ec8537b9cd67a0ff55e88240f3bdf48fba80e633994f2334fc5ebfe701ed4a8b]
pack 8c00fe576a96b4bbce3deb4a0f755e4fc32af242252175f664c20cfc8bb35238 contains 2 errors: [blob 28afe7eb6966730936558a742649de2151ba86b00eeaca2c3173657c7b287803: decrypting blob <data/28afe7eb> from 8c00fe57 failed: ciphertext verification failed unexpected pack id 02a09df7cd808eaa8c75d81bee41c8df0261ddf2322ad204b0e2b57b07408889]
pack 7ccfa4fbc681a1f81e25706e2fc01b7fa24679532ba42ed5e769b07af2532703 contains 2 errors: [blob 7299f09abe66f430cd98380177c4b8bf00269a15629cd0645834d26b4c927103: decrypting blob <data/7299f09a> from 7ccfa4fb failed: ciphertext verification failed unexpected pack id 5fafd762e4f302b0c8147861367915f15fcda5cad2472ab7a1ebc283b258b59e]
pack 1cc6e43b5ba4050ed723e5319533e9b1fa5349b1f715c8e00fa080ea54e2fbb7 contains 2 errors: [blob 4904a9747948d360a6454ecf7f349c8f248a67ea9d6b6e7fcdac26ff8ef4f9d0: decrypting blob <data/4904a974> from 1cc6e43b failed: ciphertext verification failed unexpected pack id 850beb18d0dc72e371bde20c28b40ce664a3ec17663f0bc02af37b12870a84ed]
pack 5c1d153cc57b04574538788fd13dcee7caca382c2888d4084c0ec7155d25c011 contains 2 errors: [blob 095616b34c05c7c203953f302dfa1473f78f74de4cc2346c074f6b311d39d964: decrypting blob <data/095616b3> from 5c1d153c failed: ciphertext verification failed unexpected pack id ca7088c36b8c5183da1dbae4a16bc83a687faf868b0116c68c6a23140b307eaa]
pack 4c5f4c9146f07860d61d99f5749cb3f3e1fe5f3e8de704c30c90fe76a6e46f49 contains 2 errors: [blob 82cff50ff9990efcc0900802403ecb62d8f4669a27081ebbb0afeac4b4ac7b1f: decrypting blob <data/82cff50f> from 4c5f4c91 failed: ciphertext verification failed unexpected pack id 1ed0ed686aaa9cba2f2c7463cc5bd5855c6b5aed1b1caf825d6df2ac9c7240e4]
pack 4d0bee84353bf456b7455839c9cb25e667c71e367461393c59848a27dcdb5abc contains 2 errors: [blob 0b74ccc1c27781343aaf606700bf4dbbe1e4cc63fe5768ba27647903a1bf7017: decrypting blob <data/0b74ccc1> from 4d0bee84 failed: ciphertext verification failed unexpected pack id 50731774078d2089e420854195f997ff2d361a86fb04ce7610905e8fbeb90929]
pack 8eb64ec7051ca0e0bba85aad27989638881997f73cd04fd67da1f705abe4a8e8 contains 2 errors: [blob 6ab7ef8f39598599f36ddfa54d765fc1697a3fafb824319d2de5fd903b05cf03: decrypting blob <data/6ab7ef8f> from 8eb64ec7 failed: ciphertext verification failed unexpected pack id d76ab7d5c79f71f41fa05cbba401729286b016a403e9afbbf9e92898411c49df]
pack 7e3b98ac39c7b6aafd9c794c7460d72645138576723d622456e9b6fcfdb3f83e contains 2 errors: [blob 2d309f656ecf7c03c03e64f5b7da2b9194668cef8bc715b43f3f8fb3c42d60c1: decrypting blob <data/2d309f65> from 7e3b98ac failed: ciphertext verification failed unexpected pack id cd972f41dc26c70c3fb3a748c6f86ec5cd5b3345384664cb149f51bc06e6d5b3]
pack ceb51a9c1a9fa14c3f7fa91593bb7d0c07adc5bdb0c43f9701560df63070cb8b contains 2 errors: [blob ce0473cfa22230e996ff4bee5fb7ed2ee3c84e9804a07a2176be5b73ff421acc: decrypting blob <data/ce0473cf> from ceb51a9c failed: ciphertext verification failed unexpected pack id 5e28533102daa404029f905d07a6ea73b41974c8e0cc57b09fba39fdfb831fcf]
pack bf14413f9d25af261ac92504e04e2b86e97ebdd4801abf15d7d0beeee1c56854 contains 2 errors: [blob f83c0a424bd57cc2233743131a8346a5b1a2878a5ed02250ca05376cdd34b8fa: decrypting blob <data/f83c0a42> from bf14413f failed: nonce is invalid unexpected pack id 565befe7ac4c9bc46ecb830e1fcde00aad7e20db8f998a83cb83f1fd42cfafae]
pack afcd2309393dc03c4e5bff6539d85947aa6935a2d13a5d2fca78145e800f90fb contains 2 errors: [blob 0d49f654b435fd82faa67d3399a365c7bec27165afa0d3ca11fd46d0b62f8322: decrypting blob <data/0d49f654> from afcd2309 failed: ciphertext verification failed unexpected pack id c993346a1aa636d9da61f4a3cb6f1458c03c23caead5b01b68a129a6e4020a7a]
[6:25:35] 100.00%  81729 / 81729 packs

The repository contains damaged pack files. These damaged files must be removed to repair the repository. This can be done using the following commands. Please read the troubleshooting guide at https://restic.readthedocs.io/en/stable/077_troubleshooting.html first.

restic repair packs 3191ef8002a19dc8790e88e5dba8130d3726fe0215cc36f2a33068cc6cb950eb 71923498dfb15926158e06705726517d829a9493a5c5a90e8abbe7c135bd7b31 161484e63a65d345681ce7e9334c63289928381250bf45c7956aa741315d9f58 4c5f4c9146f07860d61d99f5749cb3f3e1fe5f3e8de704c30c90fe76a6e46f49 4d0bee84353bf456b7455839c9cb25e667c71e367461393c59848a27dcdb5abc d0b0d0de9042b7be668ef687f46207b8fe917157e839c2d00d7769fd162e1ae0 43c19e0d42730145e5463d643b1fc16ce175004c4c12f2913dd3a28c9cbe90dd 34b15eb1dc61fea1b674f93f1b081ac6de94ae67c365d0f9534d66e3b4c7a1a1 15aeaf95d3ded827ae1d5b55b21e6363cad6d67cc81c70e9a14ca0b2c950729d 16a81b3e7affb66ed02490eec34386ab087a01b593ece61922200fd55eba4297 067803e995685a4b1516add892c7dad7a00e6fc15c3323ae79c94f809a357297 1b07b57b52391e630792aaf85dc13dc6086bc8f776c22f7b5e4a86c7c5d73aea afcd2309393dc03c4e5bff6539d85947aa6935a2d13a5d2fca78145e800f90fb 7e3b98ac39c7b6aafd9c794c7460d72645138576723d622456e9b6fcfdb3f83e ceb51a9c1a9fa14c3f7fa91593bb7d0c07adc5bdb0c43f9701560df63070cb8b 52d8613387a99671f3cb44f04db51914cd5ee100b077065b94c3b587414fe8f1 c5037995c681f437d311a89e6e3ab9f3115e395953f631d1f9038079284a4c5c 5b5e0692d2d828caa35c84d1d7fdd7743afee69a5eeb97fc03b49c76eb10abef 5c1d153cc57b04574538788fd13dcee7caca382c2888d4084c0ec7155d25c011 8eb64ec7051ca0e0bba85aad27989638881997f73cd04fd67da1f705abe4a8e8 bf14413f9d25af261ac92504e04e2b86e97ebdd4801abf15d7d0beeee1c56854 c33c747ab0a73fdf5c79c4a1403d960afedb84b5745e7b0d8fe4658f7223da28 f3ea14cd7332ecd1c9abb556bb4fab6339ad7213f82fa40240acf081175d4e16 c90da86133c5e3678cefbd50a4ffa14d92a1a3d3e1821ad89e9d95d1a2790fbc 1cc6e43b5ba4050ed723e5319533e9b1fa5349b1f715c8e00fa080ea54e2fbb7 0837357f89698c36a3e78279d7ef212d9151cd4a91338705f4583c53f1f10ad9 7ccfa4fbc681a1f81e25706e2fc01b7fa24679532ba42ed5e769b07af2532703 32c369f62640df226429b365d2e25f73f8ad2e22d16de8d4be69be37509fb1c7 c6cd7efe7193b6ef41d7007a34f12262b6f9f9f3f34b8ecd541e5a8995fe1eb5 679fccd2fd59e73bc13f7c6f2aab2ec3ad2b8748c19d9a39cd93c7735cef4a1e b88225fae4749651fb7eae99a5c5477f178c74e9a0187f498a58fcad1084a4a9 8c00fe576a96b4bbce3deb4a0f755e4fc32af242252175f664c20cfc8bb35238 a3c5824930c11f4ade412a49995d465cf434e1379d0a0a7adfb4e8ef1ef6b886 15ed3deb342d5ab683fea7577b49b76e5ed1dc1098955755af8ebb59831f8b53 05df8e387f2804f7f4e7ba28e6c6ab8408b2a51ac34c6347d0e79fc5867b6623 b8cf64cd61e59a9fd2fd12eea45e3765dc7f519a27d3eac2bebbeaaac601b704
restic repair snapshots --forget

Damaged pack files can be caused by backend problems, hardware problems or bugs in restic. Please open an issue at https://github.com/restic/restic/issues/new/choose for further troubleshooting!
Fatal: repository contains errors

Now, regarding NTFS, I’m using it because the drive used to hold a copy of the whole dataset instead of a proper backup, and I wanted to be able to see the contents in windows. But the plan moving forward is making the backup of one ZFS pool into another ZFS pool, then rsyncing to local drives and sending it to the cloud as well.

Taking into account everything that has happened, I would feel more comfortable formatting my external drives with ext4 and giving up on windows access (I almost always have a linux machine handy anyways).

For my beta drive I think I can just rsync the repo out of the drive, format the drive and rsync back, right?

For my alpha drive, the one that still has errors, would it make more sense to fix the errors before or after ext4?

Do you foresee any issues with migrating to ext4?

Thanks in advance! Looking forward to hearing from you.

10

Yes.

I’d slightly prefer fixing the repository afterwards, as that guarantees that the filesystem is indeed intact. Although the order depends on how long it takes you to migrate to ext4.

The restic repository just consists of a bunch of file without any special attributes, so there shouldn’t be any issues.

11

Hello,

So another progress update.

For beta drive:

  • rsynced repository out
  • reformatted to ext4
  • rsynced repository in
  • restic check --read-data → no errors
  • backup
  • restic check --read-data → no errors

For alpha drive, the damaged one:

  • rsynced repository out
  • reformatted to ext4
  • rsynced repository in
  • restic check --read-data → with errors, but less than before, log below
nohup: ignoring input
using temporary cache in /tmp/restic-check-cache-3952098635
create exclusive lock for repository
load indexes
check all packs
1219 additional files were found in the repo, which likely contain duplicate data.
This is non-critical, you can run `restic prune` to correct this.
check snapshots, trees and blobs
[0:19] 100.00%  13 / 13 snapshots
read all data
pack d0b0d0de9042b7be668ef687f46207b8fe917157e839c2d00d7769fd162e1ae0 contains 2 errors: [blob f69c51bbd3adf21663c0c3da7e47ea5b6c1cc3538c14105101c0d67f4ddfaa7c: decrypting blob <data/f69c51bb> from d0b0d0de failed: ciphertext verification failed unexpected pack id 536ca2e9f8b98da0a9900621f6ffc2f8d0037d0a7524ccc33574c20d9a6fdbce]
pack 3191ef8002a19dc8790e88e5dba8130d3726fe0215cc36f2a33068cc6cb950eb contains 2 errors: [blob 0748a4bf40a1a001cd933e2f8a37b0eb80fef29358b6d4def70d0b0e9b94ab31: decrypting blob <data/0748a4bf> from 3191ef80 failed: ciphertext verification failed unexpected pack id 732ebc641c73a55814110f31f23a86a1aaea417292727eaab101abdc185692fb]
pack 71923498dfb15926158e06705726517d829a9493a5c5a90e8abbe7c135bd7b31 contains 2 errors: [blob a96d17b7ff2b325f060ec47f3781f5fd2914499632aa540d52c5c8135517532c: decrypting blob <data/a96d17b7> from 71923498 failed: ciphertext verification failed unexpected pack id 3980629925e63046bba1b3a81a7055044d639d5c7f923a4271ee3e20a07dff8c]
pack 32c369f62640df226429b365d2e25f73f8ad2e22d16de8d4be69be37509fb1c7 contains 2 errors: [blob 3dddb4ca74459ebd3be93bfb3dde2086188d5897e7ff8e8debc527f2a4618a66: decrypting blob <data/3dddb4ca> from 32c369f6 failed: nonce is invalid unexpected pack id 479e32217520e7c7b79a88973d13f92fc2b8b58be21ec580876dd594e397b365]
pack 52d8613387a99671f3cb44f04db51914cd5ee100b077065b94c3b587414fe8f1 contains 2 errors: [blob e6fbeff2ac7d2efd7c9976a291fb4ab2d71f71669e0101ee3d530ead85a58613: decrypting blob <data/e6fbeff2> from 52d86133 failed: ciphertext verification failed unexpected pack id ce823dcf61e1841ea576007700dd16e4b74cc39cf0003d16ed00c242fb571d60]
pack c33c747ab0a73fdf5c79c4a1403d960afedb84b5745e7b0d8fe4658f7223da28 contains 2 errors: [blob 3207e176983edc9af981f551caac72b1d6041b7bc57e96e03e16c528cc891aad: decrypting blob <data/3207e176> from c33c747a failed: ciphertext verification failed unexpected pack id 1781f468366cccb7694ae05436f83f922f981879b3bf93bec1ac595791e9de3b]
pack a3c5824930c11f4ade412a49995d465cf434e1379d0a0a7adfb4e8ef1ef6b886 contains 2 errors: [blob bc69b0dab67e890a59d3376a870a950ac94a5e27cdc3d22c982c480a165b2bf9: decrypting blob <data/bc69b0da> from a3c58249 failed: ciphertext verification failed unexpected pack id 2c40edb626fc70766521b4db42ddfe90bf6ea43939ca85d0b293dfeb5b51fb2c]
pack f3ea14cd7332ecd1c9abb556bb4fab6339ad7213f82fa40240acf081175d4e16 contains 2 errors: [blob 14ec8230727fba63f6c03cbcefdb4b1e99452e203e19edaa01fd7be2518886e9: decrypting blob <data/14ec8230> from f3ea14cd failed: ciphertext verification failed unexpected pack id 3ad8e0c0e591d21423a0ab0c927686ffcb931753c5dbfb9ba15f8fc79b4a61a0]
pack 43c19e0d42730145e5463d643b1fc16ce175004c4c12f2913dd3a28c9cbe90dd contains 2 errors: [blob 0ce72796d4c99b52f7f2a03f0ee86a01a32d4d563c0c778229fab63836d3a6c3: decrypting blob <data/0ce72796> from 43c19e0d failed: ciphertext verification failed unexpected pack id e610da91bbf0115867813b0a14b540316dec7a5096bd0935682e0ac33b23126c]
pack 34b15eb1dc61fea1b674f93f1b081ac6de94ae67c365d0f9534d66e3b4c7a1a1 contains 2 errors: [blob 67cb7560417a1e57fd2435cc65c25c9374dff0dc490afbf128eb4abf2dd72e27: decrypting blob <data/67cb7560> from 34b15eb1 failed: ciphertext verification failed unexpected pack id 2fa2f4c6f6ef8a680cddb2c0a351ea05de2546def6244feb81a8187c9ece7894]
pack 05df8e387f2804f7f4e7ba28e6c6ab8408b2a51ac34c6347d0e79fc5867b6623 contains 2 errors: [blob 86e8679f7129e1e3b96fcc74a13b0eef7adb501422ece138fde52883cc74ee23: decrypting blob <data/86e8679f> from 05df8e38 failed: ciphertext verification failed unexpected pack id 105a1f0546213701482a102b460ef98a1ce5ca9950c92448b8dbc223d855ac92]
pack 15ed3deb342d5ab683fea7577b49b76e5ed1dc1098955755af8ebb59831f8b53 contains 2 errors: [blob c2d670c596c78443875529fd29912980470c20f99f224b55d6e5cad455b2c8a7: decrypting blob <data/c2d670c5> from 15ed3deb failed: ciphertext verification failed unexpected pack id 33998ad7f75b862366c5fb4687fd8803ecdfd1ddd7a9485d6dab6f5fd6fff9d5]
pack 15aeaf95d3ded827ae1d5b55b21e6363cad6d67cc81c70e9a14ca0b2c950729d contains 2 errors: [blob b9546de7716b5880ba06c02967ef52ed2fa3b81aebc22f413051f126b8583197: decrypting blob <data/b9546de7> from 15aeaf95 failed: ciphertext verification failed unexpected pack id dfd65f3089f4a9483f0cf04475b094f0f0cd6661ab6704654eeb9c34187e6375]
pack c5037995c681f437d311a89e6e3ab9f3115e395953f631d1f9038079284a4c5c contains 2 errors: [blob b171853902d4a98c97eb60f8db96d544172bc76409bc16b1a988c53f537185e9: decrypting blob <data/b1718539> from c5037995 failed: ciphertext verification failed unexpected pack id 1d678e94b1671c47003bce489750739da111dbb6e0af32933a1faf8ac36197aa]
pack 161484e63a65d345681ce7e9334c63289928381250bf45c7956aa741315d9f58 contains 2 errors: [blob da0c4312d8d2d4ffbd8b97fec609ae5ece29889e2c1a276e65de686f3befaf16: decrypting blob <data/da0c4312> from 161484e6 failed: ciphertext verification failed unexpected pack id e89b7daf827e561960a1892a9b8cdd9e71fb8443520d3515e59f6c78758b9c65]
pack 16a81b3e7affb66ed02490eec34386ab087a01b593ece61922200fd55eba4297 contains 2 errors: [blob 5899f49acd08c4c126905adc8885702e66522342b709047bac300d8bede83b26: decrypting blob <data/5899f49a> from 16a81b3e failed: ciphertext verification failed unexpected pack id 7a3a5c3676797795b7e9365c4512f3b9a9eb3f1625ba637749733938d0178a31]
pack 067803e995685a4b1516add892c7dad7a00e6fc15c3323ae79c94f809a357297 contains 2 errors: [blob c5d9059f69a622875850adcd02aa53ca37643384cb4593ffc09cb8d21feda6d2: decrypting blob <data/c5d9059f> from 067803e9 failed: ciphertext verification failed unexpected pack id cf0d8840d9d962c1f793a6aace086a6be5a40ebb24330a1f05726eded94955ce]
pack c6cd7efe7193b6ef41d7007a34f12262b6f9f9f3f34b8ecd541e5a8995fe1eb5 contains 2 errors: [blob 331a44e3a225c51440467c967cf0fecea898724d86d2ed4b56e22b46cb72d019: decrypting blob <data/331a44e3> from c6cd7efe failed: ciphertext verification failed unexpected pack id 41246b97f0f3f3d80afdcbde3b0ba7c0b6c6dafc9615de463c85421dc7be05fc]
pack 679fccd2fd59e73bc13f7c6f2aab2ec3ad2b8748c19d9a39cd93c7735cef4a1e contains 2 errors: [blob 63f384b182ee0946b3a0729d58d2ae1d917e36d25b2945b6a3703ae8b975a6c6: decrypting blob <data/63f384b1> from 679fccd2 failed: ciphertext verification failed unexpected pack id e40a32b8e69fa8f81b13e121c67dec8d8a10306536694d87f205c89df7f99088]
pack b88225fae4749651fb7eae99a5c5477f178c74e9a0187f498a58fcad1084a4a9 contains 2 errors: [blob c722e8bc16acc10d66e797b57a21162bf2fba93ab573e1bf9dfd82cf600fb57f: decrypting blob <data/c722e8bc> from b88225fa failed: ciphertext verification failed unexpected pack id 93a9352e1179c3fbe633bc8dcdf2efee548b7c391515c58ed6e1cc4517fd0f78]
pack b8cf64cd61e59a9fd2fd12eea45e3765dc7f519a27d3eac2bebbeaaac601b704 contains 2 errors: [blob d2b11592ed3d772d7b9f8e21407878e961bada4bc34154684e69728f5f69270e: decrypting blob <data/d2b11592> from b8cf64cd failed: ciphertext verification failed unexpected pack id aeaa8f5c1e35e72b33edf41f116fa1c03a1c2d15c22e6ba0e1a692807f334018]
pack 0837357f89698c36a3e78279d7ef212d9151cd4a91338705f4583c53f1f10ad9 contains 2 errors: [blob 1f3bc4912ed7962c94dfa7e9d38609bfda32e533be9f4ddacada477600bce0cb: decrypting blob <data/1f3bc491> from 0837357f failed: ciphertext verification failed unexpected pack id b2cefe8ac14b653e4a4ff59e3ef243c5f9e2a50e6ad5b8e2524dc39662191d54]
pack c90da86133c5e3678cefbd50a4ffa14d92a1a3d3e1821ad89e9d95d1a2790fbc contains 2 errors: [blob 1f0d6926430600a3cd2ae6f9e669650ac3ba5879e701de0cac9cf9b220938ade: decrypting blob <data/1f0d6926> from c90da861 failed: ciphertext verification failed unexpected pack id 665fd1cf3066fe230c14b0eb1e6fd3c57a2d5182662dad152cab06e01cd1db8a]
pack 5b5e0692d2d828caa35c84d1d7fdd7743afee69a5eeb97fc03b49c76eb10abef contains 2 errors: [blob 6aaf40065d49c857cfa725df59de058d38924449f0dcb311ed0682741a39f649: decrypting blob <data/6aaf4006> from 5b5e0692 failed: ciphertext verification failed unexpected pack id ec8537b9cd67a0ff55e88240f3bdf48fba80e633994f2334fc5ebfe701ed4a8b]
pack 1b07b57b52391e630792aaf85dc13dc6086bc8f776c22f7b5e4a86c7c5d73aea contains 2 errors: [blob aafbb6e7299ac6e69da06fef47f9cfe2dacbb1f7dfd8c95a357bdd10770560f3: decrypting blob <data/aafbb6e7> from 1b07b57b failed: ciphertext verification failed unexpected pack id f880dbce0e14fd4a10f3d13a956ca61257337cefddd4b017f55a2c62a2429f0a]
pack 4c5f4c9146f07860d61d99f5749cb3f3e1fe5f3e8de704c30c90fe76a6e46f49 contains 2 errors: [blob 82cff50ff9990efcc0900802403ecb62d8f4669a27081ebbb0afeac4b4ac7b1f: decrypting blob <data/82cff50f> from 4c5f4c91 failed: ciphertext verification failed unexpected pack id 1ed0ed686aaa9cba2f2c7463cc5bd5855c6b5aed1b1caf825d6df2ac9c7240e4]
pack 5c1d153cc57b04574538788fd13dcee7caca382c2888d4084c0ec7155d25c011 contains 2 errors: [blob 095616b34c05c7c203953f302dfa1473f78f74de4cc2346c074f6b311d39d964: decrypting blob <data/095616b3> from 5c1d153c failed: ciphertext verification failed unexpected pack id ca7088c36b8c5183da1dbae4a16bc83a687faf868b0116c68c6a23140b307eaa]
pack 7ccfa4fbc681a1f81e25706e2fc01b7fa24679532ba42ed5e769b07af2532703 contains 2 errors: [blob 7299f09abe66f430cd98380177c4b8bf00269a15629cd0645834d26b4c927103: decrypting blob <data/7299f09a> from 7ccfa4fb failed: ciphertext verification failed unexpected pack id 5fafd762e4f302b0c8147861367915f15fcda5cad2472ab7a1ebc283b258b59e]
pack 8c00fe576a96b4bbce3deb4a0f755e4fc32af242252175f664c20cfc8bb35238 contains 2 errors: [blob 28afe7eb6966730936558a742649de2151ba86b00eeaca2c3173657c7b287803: decrypting blob <data/28afe7eb> from 8c00fe57 failed: ciphertext verification failed unexpected pack id 02a09df7cd808eaa8c75d81bee41c8df0261ddf2322ad204b0e2b57b07408889]
pack 1cc6e43b5ba4050ed723e5319533e9b1fa5349b1f715c8e00fa080ea54e2fbb7 contains 2 errors: [blob 4904a9747948d360a6454ecf7f349c8f248a67ea9d6b6e7fcdac26ff8ef4f9d0: decrypting blob <data/4904a974> from 1cc6e43b failed: ciphertext verification failed unexpected pack id 850beb18d0dc72e371bde20c28b40ce664a3ec17663f0bc02af37b12870a84ed]
pack 4d0bee84353bf456b7455839c9cb25e667c71e367461393c59848a27dcdb5abc contains 2 errors: [blob 0b74ccc1c27781343aaf606700bf4dbbe1e4cc63fe5768ba27647903a1bf7017: decrypting blob <data/0b74ccc1> from 4d0bee84 failed: ciphertext verification failed unexpected pack id 50731774078d2089e420854195f997ff2d361a86fb04ce7610905e8fbeb90929]
pack 7e3b98ac39c7b6aafd9c794c7460d72645138576723d622456e9b6fcfdb3f83e contains 2 errors: [blob 2d309f656ecf7c03c03e64f5b7da2b9194668cef8bc715b43f3f8fb3c42d60c1: decrypting blob <data/2d309f65> from 7e3b98ac failed: ciphertext verification failed unexpected pack id cd972f41dc26c70c3fb3a748c6f86ec5cd5b3345384664cb149f51bc06e6d5b3]
pack 8eb64ec7051ca0e0bba85aad27989638881997f73cd04fd67da1f705abe4a8e8 contains 2 errors: [blob 6ab7ef8f39598599f36ddfa54d765fc1697a3fafb824319d2de5fd903b05cf03: decrypting blob <data/6ab7ef8f> from 8eb64ec7 failed: ciphertext verification failed unexpected pack id d76ab7d5c79f71f41fa05cbba401729286b016a403e9afbbf9e92898411c49df]
pack ceb51a9c1a9fa14c3f7fa91593bb7d0c07adc5bdb0c43f9701560df63070cb8b contains 2 errors: [blob ce0473cfa22230e996ff4bee5fb7ed2ee3c84e9804a07a2176be5b73ff421acc: decrypting blob <data/ce0473cf> from ceb51a9c failed: ciphertext verification failed unexpected pack id 5e28533102daa404029f905d07a6ea73b41974c8e0cc57b09fba39fdfb831fcf]
pack bf14413f9d25af261ac92504e04e2b86e97ebdd4801abf15d7d0beeee1c56854 contains 2 errors: [blob f83c0a424bd57cc2233743131a8346a5b1a2878a5ed02250ca05376cdd34b8fa: decrypting blob <data/f83c0a42> from bf14413f failed: nonce is invalid unexpected pack id 565befe7ac4c9bc46ecb830e1fcde00aad7e20db8f998a83cb83f1fd42cfafae]
pack afcd2309393dc03c4e5bff6539d85947aa6935a2d13a5d2fca78145e800f90fb contains 2 errors: [blob 0d49f654b435fd82faa67d3399a365c7bec27165afa0d3ca11fd46d0b62f8322: decrypting blob <data/0d49f654> from afcd2309 failed: ciphertext verification failed unexpected pack id c993346a1aa636d9da61f4a3cb6f1458c03c23caead5b01b68a129a6e4020a7a]
[3:09:10] 100.00%  81729 / 81729 packs

The repository contains damaged pack files. These damaged files must be removed to repair the repository. This can be done using the following commands. Please read the troubleshooting guide at https://restic.readthedocs.io/en/stable/077_troubleshooting.html first.

restic repair packs 4c5f4c9146f07860d61d99f5749cb3f3e1fe5f3e8de704c30c90fe76a6e46f49 1cc6e43b5ba4050ed723e5319533e9b1fa5349b1f715c8e00fa080ea54e2fbb7 bf14413f9d25af261ac92504e04e2b86e97ebdd4801abf15d7d0beeee1c56854 ceb51a9c1a9fa14c3f7fa91593bb7d0c07adc5bdb0c43f9701560df63070cb8b 15aeaf95d3ded827ae1d5b55b21e6363cad6d67cc81c70e9a14ca0b2c950729d 16a81b3e7affb66ed02490eec34386ab087a01b593ece61922200fd55eba4297 5b5e0692d2d828caa35c84d1d7fdd7743afee69a5eeb97fc03b49c76eb10abef 7e3b98ac39c7b6aafd9c794c7460d72645138576723d622456e9b6fcfdb3f83e 71923498dfb15926158e06705726517d829a9493a5c5a90e8abbe7c135bd7b31 161484e63a65d345681ce7e9334c63289928381250bf45c7956aa741315d9f58 c6cd7efe7193b6ef41d7007a34f12262b6f9f9f3f34b8ecd541e5a8995fe1eb5 c90da86133c5e3678cefbd50a4ffa14d92a1a3d3e1821ad89e9d95d1a2790fbc 8c00fe576a96b4bbce3deb4a0f755e4fc32af242252175f664c20cfc8bb35238 8eb64ec7051ca0e0bba85aad27989638881997f73cd04fd67da1f705abe4a8e8 52d8613387a99671f3cb44f04db51914cd5ee100b077065b94c3b587414fe8f1 c33c747ab0a73fdf5c79c4a1403d960afedb84b5745e7b0d8fe4658f7223da28 b8cf64cd61e59a9fd2fd12eea45e3765dc7f519a27d3eac2bebbeaaac601b704 0837357f89698c36a3e78279d7ef212d9151cd4a91338705f4583c53f1f10ad9 5c1d153cc57b04574538788fd13dcee7caca382c2888d4084c0ec7155d25c011 3191ef8002a19dc8790e88e5dba8130d3726fe0215cc36f2a33068cc6cb950eb a3c5824930c11f4ade412a49995d465cf434e1379d0a0a7adfb4e8ef1ef6b886 34b15eb1dc61fea1b674f93f1b081ac6de94ae67c365d0f9534d66e3b4c7a1a1 b88225fae4749651fb7eae99a5c5477f178c74e9a0187f498a58fcad1084a4a9 d0b0d0de9042b7be668ef687f46207b8fe917157e839c2d00d7769fd162e1ae0 05df8e387f2804f7f4e7ba28e6c6ab8408b2a51ac34c6347d0e79fc5867b6623 067803e995685a4b1516add892c7dad7a00e6fc15c3323ae79c94f809a357297 1b07b57b52391e630792aaf85dc13dc6086bc8f776c22f7b5e4a86c7c5d73aea c5037995c681f437d311a89e6e3ab9f3115e395953f631d1f9038079284a4c5c 679fccd2fd59e73bc13f7c6f2aab2ec3ad2b8748c19d9a39cd93c7735cef4a1e 7ccfa4fbc681a1f81e25706e2fc01b7fa24679532ba42ed5e769b07af2532703 4d0bee84353bf456b7455839c9cb25e667c71e367461393c59848a27dcdb5abc 32c369f62640df226429b365d2e25f73f8ad2e22d16de8d4be69be37509fb1c7 f3ea14cd7332ecd1c9abb556bb4fab6339ad7213f82fa40240acf081175d4e16 43c19e0d42730145e5463d643b1fc16ce175004c4c12f2913dd3a28c9cbe90dd 15ed3deb342d5ab683fea7577b49b76e5ed1dc1098955755af8ebb59831f8b53 afcd2309393dc03c4e5bff6539d85947aa6935a2d13a5d2fca78145e800f90fb
restic repair snapshots --forget

Damaged pack files can be caused by backend problems, hardware problems or bugs in restic. Please open an issue at https://github.com/restic/restic/issues/new/choose for further troubleshooting!
Fatal: repository contains errors

So now that the situation is stable, should I follow the instructions in the log (prune, repair packs, repair snapshots --forget) or some special instructions?

Looking forward to your guidance.

Thanks in advance!

12

Follow the steps suggested by the check command. If you want, you can rerun your backup jobs between repair packs and repair snapshots. That should minimize the amount of data lost.

13

All done!

I ran repair packs:

nohup: ignoring input
saving backup copies of pack files to current folder
salvaging intact data from specified pack files
failed to load blob f69c51bbd3adf21663c0c3da7e47ea5b6c1cc3538c14105101c0d67f4ddfaa7c: decrypting blob <data/f69c51bb> from d0b0d0de failed: ciphertext verification failed
failed to load blob 0748a4bf40a1a001cd933e2f8a37b0eb80fef29358b6d4def70d0b0e9b94ab31: decrypting blob <data/0748a4bf> from 3191ef80 failed: ciphertext verification failed
failed to load blob a96d17b7ff2b325f060ec47f3781f5fd2914499632aa540d52c5c8135517532c: decrypting blob <data/a96d17b7> from 71923498 failed: ciphertext verification failed
failed to load blob e6fbeff2ac7d2efd7c9976a291fb4ab2d71f71669e0101ee3d530ead85a58613: decrypting blob <data/e6fbeff2> from 52d86133 failed: ciphertext verification failed
failed to load blob 3dddb4ca74459ebd3be93bfb3dde2086188d5897e7ff8e8debc527f2a4618a66: decrypting blob <data/3dddb4ca> from 32c369f6 failed: nonce is invalid
failed to load blob bc69b0dab67e890a59d3376a870a950ac94a5e27cdc3d22c982c480a165b2bf9: decrypting blob <data/bc69b0da> from a3c58249 failed: ciphertext verification failed
failed to load blob 14ec8230727fba63f6c03cbcefdb4b1e99452e203e19edaa01fd7be2518886e9: decrypting blob <data/14ec8230> from f3ea14cd failed: ciphertext verification failed
failed to load blob 3207e176983edc9af981f551caac72b1d6041b7bc57e96e03e16c528cc891aad: decrypting blob <data/3207e176> from c33c747a failed: ciphertext verification failed
failed to load blob 0ce72796d4c99b52f7f2a03f0ee86a01a32d4d563c0c778229fab63836d3a6c3: decrypting blob <data/0ce72796> from 43c19e0d failed: ciphertext verification failed
failed to load blob 67cb7560417a1e57fd2435cc65c25c9374dff0dc490afbf128eb4abf2dd72e27: decrypting blob <data/67cb7560> from 34b15eb1 failed: ciphertext verification failed
failed to load blob 86e8679f7129e1e3b96fcc74a13b0eef7adb501422ece138fde52883cc74ee23: decrypting blob <data/86e8679f> from 05df8e38 failed: ciphertext verification failed
failed to load blob b171853902d4a98c97eb60f8db96d544172bc76409bc16b1a988c53f537185e9: decrypting blob <data/b1718539> from c5037995 failed: ciphertext verification failed
failed to load blob b9546de7716b5880ba06c02967ef52ed2fa3b81aebc22f413051f126b8583197: decrypting blob <data/b9546de7> from 15aeaf95 failed: ciphertext verification failed
failed to load blob c2d670c596c78443875529fd29912980470c20f99f224b55d6e5cad455b2c8a7: decrypting blob <data/c2d670c5> from 15ed3deb failed: ciphertext verification failed
failed to load blob 5899f49acd08c4c126905adc8885702e66522342b709047bac300d8bede83b26: decrypting blob <data/5899f49a> from 16a81b3e failed: ciphertext verification failed
failed to load blob 331a44e3a225c51440467c967cf0fecea898724d86d2ed4b56e22b46cb72d019: decrypting blob <data/331a44e3> from c6cd7efe failed: ciphertext verification failed
failed to load blob da0c4312d8d2d4ffbd8b97fec609ae5ece29889e2c1a276e65de686f3befaf16: decrypting blob <data/da0c4312> from 161484e6 failed: ciphertext verification failed
failed to load blob c5d9059f69a622875850adcd02aa53ca37643384cb4593ffc09cb8d21feda6d2: decrypting blob <data/c5d9059f> from 067803e9 failed: ciphertext verification failed
failed to load blob 63f384b182ee0946b3a0729d58d2ae1d917e36d25b2945b6a3703ae8b975a6c6: decrypting blob <data/63f384b1> from 679fccd2 failed: ciphertext verification failed
failed to load blob 1f3bc4912ed7962c94dfa7e9d38609bfda32e533be9f4ddacada477600bce0cb: decrypting blob <data/1f3bc491> from 0837357f failed: ciphertext verification failed
failed to load blob d2b11592ed3d772d7b9f8e21407878e961bada4bc34154684e69728f5f69270e: decrypting blob <data/d2b11592> from b8cf64cd failed: ciphertext verification failed
failed to load blob c722e8bc16acc10d66e797b57a21162bf2fba93ab573e1bf9dfd82cf600fb57f: decrypting blob <data/c722e8bc> from b88225fa failed: ciphertext verification failed
failed to load blob 1f0d6926430600a3cd2ae6f9e669650ac3ba5879e701de0cac9cf9b220938ade: decrypting blob <data/1f0d6926> from c90da861 failed: ciphertext verification failed
failed to load blob aafbb6e7299ac6e69da06fef47f9cfe2dacbb1f7dfd8c95a357bdd10770560f3: decrypting blob <data/aafbb6e7> from 1b07b57b failed: ciphertext verification failed
failed to load blob 6aaf40065d49c857cfa725df59de058d38924449f0dcb311ed0682741a39f649: decrypting blob <data/6aaf4006> from 5b5e0692 failed: ciphertext verification failed
failed to load blob 4904a9747948d360a6454ecf7f349c8f248a67ea9d6b6e7fcdac26ff8ef4f9d0: decrypting blob <data/4904a974> from 1cc6e43b failed: ciphertext verification failed
failed to load blob 82cff50ff9990efcc0900802403ecb62d8f4669a27081ebbb0afeac4b4ac7b1f: decrypting blob <data/82cff50f> from 4c5f4c91 failed: ciphertext verification failed
failed to load blob 095616b34c05c7c203953f302dfa1473f78f74de4cc2346c074f6b311d39d964: decrypting blob <data/095616b3> from 5c1d153c failed: ciphertext verification failed
failed to load blob 7299f09abe66f430cd98380177c4b8bf00269a15629cd0645834d26b4c927103: decrypting blob <data/7299f09a> from 7ccfa4fb failed: ciphertext verification failed
failed to load blob 28afe7eb6966730936558a742649de2151ba86b00eeaca2c3173657c7b287803: decrypting blob <data/28afe7eb> from 8c00fe57 failed: ciphertext verification failed
failed to load blob 0b74ccc1c27781343aaf606700bf4dbbe1e4cc63fe5768ba27647903a1bf7017: decrypting blob <data/0b74ccc1> from 4d0bee84 failed: ciphertext verification failed
failed to load blob 2d309f656ecf7c03c03e64f5b7da2b9194668cef8bc715b43f3f8fb3c42d60c1: decrypting blob <data/2d309f65> from 7e3b98ac failed: ciphertext verification failed
failed to load blob 6ab7ef8f39598599f36ddfa54d765fc1697a3fafb824319d2de5fd903b05cf03: decrypting blob <data/6ab7ef8f> from 8eb64ec7 failed: ciphertext verification failed
failed to load blob ce0473cfa22230e996ff4bee5fb7ed2ee3c84e9804a07a2176be5b73ff421acc: decrypting blob <data/ce0473cf> from ceb51a9c failed: ciphertext verification failed
failed to load blob 0d49f654b435fd82faa67d3399a365c7bec27165afa0d3ca11fd46d0b62f8322: decrypting blob <data/0d49f654> from afcd2309 failed: ciphertext verification failed
failed to load blob f83c0a424bd57cc2233743131a8346a5b1a2878a5ed02250ca05376cdd34b8fa: decrypting blob <data/f83c0a42> from bf14413f failed: nonce is invalid
[0:06] 100.00%  36 / 36 pack files
rebuilding index
[0:02] 100.00%  75 / 75 indexes processed
[0:00] 100.00%  75 / 75 old indexes deleted
removing salvaged pack files
[0:00] 100.00%  36 / 36 files deleted

Use `restic repair snapshots --forget` to remove the corrupted data blobs from all snapshots

Then I ran my usual backup:

nohup: ignoring input
open repository
lock repository
load index files
using parent snapshot 0c8e447b
start scan on [/gringotts/NAS /gringotts/ON /gringotts/MH]
start backup on [/gringotts/NAS /gringotts/ON /gringotts/MH]
scan finished in 4.029s: 184189 files, 615.651 GiB
error: parts of /gringotts/NAS/syncthing/Camera/Camera/20240630_144525.mp4 not found in the repository index; storing the file again
error: parts of /gringotts/NAS/syncthing/Camera/Camera/20240630_151838.mp4 not found in the repository index; storing the file again
error: parts of /gringotts/NAS/syncthing/Camera/Camera/20240630_153057.mp4 not found in the repository index; storing the file again

Files:        3909 new,    12 changed, 180268 unmodified
Dirs:           52 new,    42 changed, 17139 unmodified
Data Blobs:   7953 new
Tree Blobs:     90 new
Added to the repo: 10.433 GiB

processed 184189 files, 615.651 GiB in 7:31
snapshot 6af3103a saved
Warning: failed to read all source data during backup

Then I ran repair snapshots --forget:

nohup: ignoring input

snapshot 6c0fe2e3 of [/gringotts/NAS /gringotts/ON /gringotts/MH] at 2024-04-07 18:19:12.535264576 -0600 CST by rurushu@lilchewchew2-0

snapshot 44d4c113 of [/gringotts/NAS /gringotts/ON /gringotts/MH] at 2024-04-14 20:52:54.663095038 -0600 CST by rurushu@lilchewchew2-0

snapshot 1cc4ec51 of [/gringotts/NAS /gringotts/ON /gringotts/MH] at 2024-02-12 22:14:14.80754614 -0600 CST by rurushu@lilchewchew2-0

snapshot 06fa90ff of [/gringotts/NAS /gringotts/ON /gringotts/MH] at 2024-10-31 15:58:21.919835271 -0600 CST by rurushu@lilchewchew2-0

snapshot f02ba6c1 of [/gringotts/NAS /gringotts/ON /gringotts/MH] at 2024-07-18 13:49:40.428641974 -0600 CST by rurushu@lilchewchew2-0

snapshot 36b1c460 of [/gringotts/NAS /gringotts/ON /gringotts/MH] at 2024-07-01 16:39:46.719786653 -0600 CST by rurushu@lilchewchew2-0

snapshot 02b21571 of [/gringotts/NAS /gringotts/ON /gringotts/MH] at 2024-05-14 14:10:27.994787563 -0600 CST by rurushu@lilchewchew2-0

snapshot 3b1346ca of [/gringotts/NAS /gringotts/ON /gringotts/MH] at 2024-04-20 20:05:33.279888207 -0600 CST by rurushu@lilchewchew2-0

snapshot 5caa6ea2 of [/gringotts/NAS /gringotts/ON /gringotts/MH] at 2024-06-05 19:43:59.924323365 -0600 CST by rurushu@lilchewchew2-0

snapshot 6af3103a of [/gringotts/NAS /gringotts/ON /gringotts/MH] at 2024-10-31 15:25:05.76862212 -0600 CST by rurushu@lilchewchew2-0

snapshot 0c8e447b of [/gringotts/NAS /gringotts/ON /gringotts/MH] at 2024-08-20 01:32:07.52550889 -0600 CST by rurushu@lilchewchew2-0

snapshot 890f10cd of [/gringotts/NAS /gringotts/ON /gringotts/MH] at 2024-06-09 14:55:55.243016392 -0600 CST by rurushu@lilchewchew2-0

snapshot 65b13d3a of [/gringotts/NAS /gringotts/ON /gringotts/MH] at 2024-05-05 17:49:39.777224611 -0600 CST by rurushu@lilchewchew2-0

snapshot 39ed1f92 of [/gringotts/NAS /gringotts/ON /gringotts/MH] at 2024-07-29 16:21:26.374321613 -0600 CST by rurushu@lilchewchew2-0

snapshot 52064ae1 of [/gringotts/NAS /gringotts/ON /gringotts/MH] at 2024-03-19 21:42:17.30182222 -0600 CST by rurushu@lilchewchew2-0

no snapshots were modified

Then finally I ran check --read-data:

nohup: ignoring input
using temporary cache in /tmp/restic-check-cache-3417075369
create exclusive lock for repository
load indexes
check all packs
1219 additional files were found in the repo, which likely contain duplicate data.
This is non-critical, you can run `restic prune` to correct this.
check snapshots, trees and blobs
[0:20] 100.00%  15 / 15 snapshots
read all data
[3:22:22] 100.00%  83839 / 83839 packs
no errors were found

Seems like it’s done, right? If so, would it be wise for me to summarize the issue and the solution and mark that post as the solution?

Thank you so much for all the help!

14

Yes, check --read-data is happy so your repository is definitely fine.

You can if you want. Although most of the troubleshooting steps except replacing / repairing the filesystem should (hopefully) already be documented at Troubleshooting — restic 0.17.2 documentation .

15

Hello,

FYI @damoclark @MichaelEischer just for the sake of being thorough, I tried a restic check --read-data on the alpha (now ext4) external drive with no external power, without the externally powered USB hub, trying to reproduce the disconnects that happened every single time on the formerly NTFS drive. I could not reproduce the issue, I think this means the whole problem came just from the NTFS implementation on Linux.

So, to summarize this whole thread:

Problem

The NTFS on the drive the repository was stored in, was pretty damaged and had a few folders that could not be read, leaving the repository in a broken state.

This damage to the filesystem was likely originated by multiple disconnects/resets of the drive while operating, these could be observed during restic check --read-data command in the dmesg output, but likely occurred during backups as well.

It seemed like this was a hardware issue, it was consistently reproducible but it stopped happening when using an externally powered USB hub in between the drive and the computer, which further hinted at a hardware issue, however, after changing the partition format to ext4, the error cannot be reproduced under the original circumstances, maybe hinting at a problem with NTFS itself instead of the hardware.

Solution

After thoroughly checking that the hardware was not damaged (smartctl tests, badblocks, dd, ddrescue, etc.), the solution was to change the filesystem on the drive from NTFS to ext4 which got rid of all error messages in dmesg.

Once the drive was stable enough to work on, the steps on the troubleshooting guide worked like a charm.

Conclusion

The NTFS implementation on Linux is brittle at best, the utilities to check/repair the FS are not good enough and you end up relying on a windows machine to fix the issue. I wouldn’t recommend using NTFS again for this.

Thanks for all the help and for sharing your expertise with me to help me solve this problem. I’m very happy with the interactions we had and thankful to the community.

1 Like