Hello everyone.
After a prune, index is corrupted.
Before the prune, I checked the repository and had just some unused packs.
The backend is a local folder, under Windows.
I’m using restic 0.16.4.
Now I’m using repair index --read-all-packs (without it, it didn’t work). I hope this works.
Maybe there is a bug in restic?
=== Prune ===
repository 8ac455d3 opened (version 2, compression level auto)
loading indexes...
[0:02] 100.00% 70 / 70 index files loaded
loading all snapshots...
finding data that is still in use for 179 snapshots
[0:31] 100.00% 179 / 179 snapshots
searching used packs...
collecting packs for deletion and repacking
[0:16] 100.00% 99835 / 99835 packs processed
to repack: 597 blobs / 3.060 MiB
this removes: 0 blobs / 0 B
to delete: 0 blobs / 9.921 GiB
total prune: 0 blobs / 9.921 GiB
remaining: 2218835 blobs / 1.615 TiB
unused size after prune: 272.045 MiB (0.02% of remaining size)
deleting unreferenced packs
[0:26] 100.00% 598 / 598 files deleted
repacking packs
[0:01] 100.00% 19 / 19 packs repacked
rebuilding index
[0:03] 100.00% 99818 / 99818 packs processed
deleting obsolete index files
[0:03] 100.00% 70 / 70 files deleted
removing 19 old packs
[0:01] 100.00% 19 / 19 files deleted
done
=== Check ===
using temporary cache in C:\Users\---\AppData\Local\Temp\restic-check-cache-3545433617
repository 8ac455d3 opened (version 2, compression level auto)
created new cache in C:\Users\---\AppData\Local\Temp\restic-check-cache-3545433617
create exclusive lock for repository
load indexes
[0:00] 100.00% 15 / 15 index files loaded
error: error loading index eab41e76b30ab031028685360e5608489f0a61d37f20cf6c5a886d2dfb3649a2: invalid data returned
error: error loading index 1522139098efb855c96e9346a62af967123b8346e0dfbc31e17d80588bc05eae: invalid data returned
error: error loading index 7486915eb01b33aa7e2514aa26c70708b6ee0f9f8835f93f336c44a9c0b453bb: invalid data returned
error: error loading index 2012ab55b6f78218d8bf106468245226df089c36b9e816e63d916605723eaf8f: invalid data returned
error: error loading index 3904f511bb5343278e775d1d87c1d8358d6bded8616d192b9f84a0f52a672db5: invalid data returned
error: error loading index 3bb81cb57b964d17b3570d9fc334e8ca9fe95cb353a61fe9f2867e059840864f: invalid data returned
error: error loading index 53fcda9ee12186cdab79f545c9594aeffe4c06b8fd9e72788a3a1c7e4b7bdf2e: invalid data returned
error: error loading index b5986d75e405154155c64e29ea17c4a299625e0d788968eebed26a6aef3ce2c1: invalid data returned
error: error loading index 382c8983ee167be15b775e3b91d2d3c4eaa8a1041444e4c52303bff3b68e09d8: invalid data returned
error: error loading index 88bd36e32894730eda98d1e44ece66dc6e6d44bedfa6274685ed448b294e3621: invalid data returned
error: error loading index 8cfe01da55c3af2e3f50f26e117c80e63ba607d4e0941c02248cf40b5abe9677: invalid data returned
error: error loading index c5984ddfadf76bb48799403ebb3a9a253368e80a1dea462d04b4ff72dc271eb9: invalid data returned
error: error loading index e2fdf1c1ffdbf751e80a528f77bc7a4da7d34c2fb3616c32329b8c0767c168e4: invalid data returned
error: error loading index c213b70e771419e8da1c78c9c0d5b386d5695bdd2eb34a68feceb4cef728ea5f: invalid data returned
error: error loading index f81cd4e885f3648c7645cd3210bb938538cdf74c1b9a84173898bff95aa13bc7: invalid data returned
Fatal: LoadIndex returned errors
restic repair index --read-all-packs finished without errors, but the subsequent check still return errors.
using temporary cache in C:\Users\---\AppData\Local\Temp\restic-check-cache-393379467
repository 8ac455d3 opened (version 2, compression level auto)
created new cache in C:\Users\---\AppData\Local\Temp\restic-check-cache-393379467
create exclusive lock for repository
load indexes
[0:00] 100.00% 15 / 15 index files loaded
error: error loading index 32d09cec6c379d5e829d3bb3dfdc8c65f0de88813338fa57cfce0e8c41bbb789: invalid data returned
error: error loading index 1b6410f7f40969ab62cc69a50c6b9fd72c3d72e48081a9c99eea0b70857de209: invalid data returned
error: error loading index 5a6d81ef66a3101534fa18992b00f7af30aeb42308fab32b45f5d16cb7156e4b: invalid data returned
error: error loading index 394372a21366d92d6852d7d5dfe3fe4f4ff55ed271edc4c595f0bb0941868c6d: invalid data returned
error: error loading index 4baf584e1c2b359ba939266c858b9f9dda765f85d3e484c847e306d4a82a85b2: invalid data returned
error: error loading index 1cfdc6c7a5866a80b1f5099e9b0c1b28524a74b43ea8444e50526ff316e09068: invalid data returned
error: error loading index 812a85b94e11c7f6d72f9156f813aaa3b08cdcf0b4a759afa0d986c386f9407a: invalid data returned
error: error loading index eb3aa77067c436f90c6cff92a046e4f016b9b57523f1b9d7f9c78b053e070af3: invalid data returned
error: error loading index b438f5fc17ea81f82cd29e7c7fdfdfa3d197dbf031a01c62ba7ea512777a1692: invalid data returned
error: error loading index dca514bf63ebafb9496fa344e7073d127123b90f7432aeef0bd8e1c395e7257f: invalid data returned
error: error loading index 69b0e5f48cec207d23207ad58278c9f205ed404cf8e675e35e9c63f1d787d08e: invalid data returned
error: error loading index 3153ab78e8da21691293e6bd10cccbc1bd410bf0dba5d427c5742ab74c0413f2: invalid data returned
error: error loading index 8b4eaec0e2565081894aa2d637421e6c8171420bde569ad2878834dc26786ac4: invalid data returned
error: error loading index 5b9124ba7b97fcf6df14bc11ce7cce129a8bb02f8e9ccdb044ced23224a9e4d7: invalid data returned
error: error loading index efefeeff0db1ae5152629677e1fe81d2eec6c35952e8e3f62e89d061f99f9636: invalid data returned
Fatal: LoadIndex returned errors
I used this script to check data:
stdbuf -i0 -o0 -e0 find [backup repository] -mtime -1 -type f | parallel sha256sum {} | tee check_last_data.log
echo Errors:
awk '1 {name = $2; sub(/.*\//, "", name); if ($1 != name) {print}}' check_last_data.log
This found corrupted data packs. I will delete them, run backup and rebuild index, then update this thread.
I did the very same check before prune and everything was well, so I’m sure that the corruption happened during pruining.
I’m almost sure this happened once before, but in that occasion I blamed my Windows installation which suffered frequent BSODs. This is a new computer with a fresh installation.
The index loading errors are very, very strange. check complains that 15 out of a total of 15 indexes are invalid. restic repair index --read-all-packs apparently rewrote the indexes as the filenames of the index files have changed.
Please run sha256sum for the index files. Do they match their filename? Does an older restic version e.g. 0.15.2 work?
No, index files hash do not match, not a single one. Three packs are also broken, and some snapshots.
I don’t have the slightest clue how that could happen. The relevant part of the restic code is rather straightforward and hasn’t changed much recently. Did you try whether an older restic version works? Is maybe some security software interfering with restic?
What do you mean by using older versions? Prune or rebuild index? With which version?
As for security software… For each and every index file, twice? And only while pruning, not during backup? It seems strange to me.
Please try whether restic 0.15.2 works for you or not. As the index is broken, you’ll have to rebuild it first in any case. Afterwards prune hopefully works.
It is indeed strange, but so far you seem to be the only one with this particular issue.
restic 0.15.2 seems to work.
Today I don’t have the original data to “repair” the database, I will follow up in a week. Pruning won’t work in this state.
E:\--->restic_0.15.2_windows_amd64.exe rebuild-index
repository 8ac455d3 opened (version 2, compression level auto)
loading indexes...
getting pack files to read...
removing not found pack file c4ac1f37da5ea2945ac3a06dc0facb893262d61829dc9eac95cc42e7a8acc27a
removing not found pack file 19e917f4ad25628209490ccba31e372715b0dd1d17c9e5527b88c46110929561
rebuilding index
[0:05] 100.00% 99816 / 99816 packs processed
deleting obsolete index files
[0:00] 100.00% 16 / 16 files deleted
done
E:\--->restic_0.15.2_windows_amd64.exe check
using temporary cache in C:\Users\---\AppData\Local\Temp\restic-check-cache-3831139570
repository 8ac455d3 opened (version 2, compression level auto)
created new cache in C:\Users\---\AppData\Local\Temp\restic-check-cache-3831139570
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
error: invalid data returned
error: invalid data returned
error: invalid data returned
error: invalid data returned
error: invalid data returned
error for tree ff06610c:
id ff06610c8ec8e603adb518c17718710bc7e77bf30607c7be988372bdd62c699f not found in repository
error for tree f612641d:
id f612641d9656aa779f971138fe5b680ec7ae882d75fae0702315567b54fc446d not found in repository
error for tree 1263557a:
id 1263557a8abcdf1496003033f42058c54c5bbd481deb742344014c579cb99208 not found in repository
error for tree f2cdb408:
id f2cdb40801a01814aa1fdeb685d4212775ef5abd62de8fbc508b9cc36a625333 not found in repository
error for tree bb7a73bb:
id bb7a73bbc363b531d64241c08b38d745df81d2a6cf2d5765a5dafaf5fd65170c not found in repository
error for tree 7fa6e12d:
id 7fa6e12dc1f0355eb4dddb133d016583880d94a26d49915b70e0f7e23aae8d45 not found in repository
error for tree 6e91e18e:
id 6e91e18ea9f76432bd2fff7dd7e18e8bb186b1814e565a8399082b84b16d033d not found in repository
error for tree 0ba9a32b:
id 0ba9a32b3d6ececc48914912611010c221f599aeda9229c97b23ec92b21ae30b not found in repository
error for tree 62f3f83e:
id 62f3f83eb8618a1b91400aa36be583f7df9f8f5679d15924bff7d4de26123291 not found in repository
error for tree b53eef87:
id b53eef878bbafa36a376166090287bdf08f1d14e4474245aa3834f13dbf484c6 not found in repository
error for tree 7d596fca:
id 7d596fca54d092aa9a26dc143c1104d4cc802669e5e4a20b151ead2dd6aac7b5 not found in repository
error for tree 44dc82ae:
id 44dc82ae76124757272a97d69bf43b19a874a7beb9762c62344424bf1a38bf3d not found in repository
error for tree 56b92aee:
id 56b92aeebcf89ccf70a2f3386404ec7022f918d81d7288db58afa74b89b1a4fb not found in repository
error for tree dae21d13:
id dae21d136053692323e61c2a2b0c60a14a5ebe3c9a52d937c0ed6319225787f7 not found in repository
error for tree 766eefac:
id 766eefacaed67176b006d6056b6ab8898435d90e57b1961ce096b3de361bdd80 not found in repository
error for tree a43d1b11:
id a43d1b11f50c43aef81f1409d04204d0a10fbd6701fcf18a6ed60de5607bd336 not found in repository
error for tree fdbf0dea:
id fdbf0dea8f7d4a8d60f1c09f6d7c1a123be1f8bd5ed4255bb1254adcc8343805 not found in repository
error for tree 7686fc0c:
id 7686fc0c8826eb44c8e66aedb61e2603d1e5bc9aeeca16e8433950b6e4f1c9a6 not found in repository
error for tree 9200e774:
id 9200e7746911a1393011600f878351acb50bf30ba5ac21f883cc48d8e4b46094 not found in repository
error for tree f6e01bd3:
id f6e01bd3146745ec33382c27c1b4163bb2a1384f937667d7f7b4d276e85015c8 not found in repository
error for tree f9d3b0f6:
id f9d3b0f66f4393b71e00e88d62e133b11b32496aa05b21e486a45c3eafb0e22c not found in repository
error for tree 7194aa97:
id 7194aa97455becae80f67736958660fa537cceeaede5a2ba7c11aa08da75bf84 not found in repository
error for tree da858c8c:
id da858c8c7b303a847bfb4eda86a0f1fe401290a2878e898756578aea6ef6a247 not found in repository
error for tree 2186b3a7:
id 2186b3a7321500b0c6c5de6a26b23657e1c06df5f661457884a0a43b823f6aa5 not found in repository
error for tree cba5ce0b:
id cba5ce0b3ea303d35bc16588deac8541c4fadd1820289d2e3e2f7b2ffdadc693 not found in repository
error for tree a7c6d105:
id a7c6d1051ab44582e413ee30c41d759f1633e6254a7d72f95bc16629bc11a2b3 not found in repository
[0:29] 100.00% 179 / 179 snapshots
Fatal: repository contains errors
Interestingly, restic rebuild-index --read-all-packs with 0.15.2 took less than half the time restick repair index --read-all-packs with 0.16.4 (about 60m vs. 150m). Subsequent check loaded indexes (but found several broken snapshots).
After using 0.16.4 repair snapshots --forget, the subsequent check correctly read indexes.
I run the backup once again with 0.16.4, and a significant part of data where lost. I checked sha256sum of modified data, finding no problem. Also, the check found no problem.
I will not try prune again with 0.16.4 for this repository for now, not until I can backup it just for tests.
I am using restic 0.16.2 compiled with go1.22rc2 on linux/amd64 on Fedora 40 and today I experienced similar issue,
[0:06] 100.00% 10 / 10 index files loaded
error: error loading index 729deaad2c91c2ef0c5d496fa4fd6c8736e654f8b0045c7c26ad2158e94244f9: invalid data returned
error: error loading index 3f9997a48c7fd49520b0beeef745db2d5f76612984812757c921f8f3c2d3c075: invalid data returned
error: error loading index 2961912793d5e2ed91466db8dddef1236472260740ff36aef53e35faaf8ede2f: invalid data returned
error: error loading index abd352ee55c42fe6dfbc7ff1e1d116470501d75cbaa4461a5db5120b08e94303: invalid data returned
Fatal: LoadIndex returned errors
Hi @Restic_User
You haven’t provided us much detail to go on.
Firstly, I’d suggest upgrading to latest version of Restic, which at this point in time is 0.17.1. There have been many updates since 0.16.2.
Then, you should step through the Troubleshooting steps outlined in the manual. Let us know what you learn.
1 Like
Thanks for the reply. I was just reporting the same behavior I noticed, and I am using it to back up to a cloud storage. Fedora has only 0.16.2 to their repository, and I couldn’t figure out how would I install the 0.17.1. However, I followed the troubleshooting steps (restic repair index --read-all-packs) and it is all good for now.
Using your OS package manager remove installed restic, for Fedora I think it will be:
sudo dnf remove restic
Download latest restic binary from github releases page. Most likely you need restic_0.17.1_linux_amd64.bz2 .
Decompress it, give it excecutable permissions, put wherever you prefer and start using. Next time new restic is released you can simply run restic self-update (do not do it on binary installed by package manager though)
Restic is single file binary without any extra dependencies. There is absolutely no reason to use outdated versions provided by some linux distributions.
If you care contact you distro maintainers and try to improve situation with restic. It is outdates as simply nobody cares and is willing to change it. Situation differs for different distros - for example in my Suse Tumbleweed restic is always updated few days after release.
1 Like
Thank you so much for the instructions. I’m new to Linux, and I’m learning every day.
I will try, but it will probably not be updated as much as rolling releases like Arch or TW. Anyway, I appreciate your help greatly.
Update:
$ restic version
restic 0.17.1 compiled with go1.23.1 on linux/amd64
Judging from the “invalid data returned” errors, I’d assume that the cloud storage has corrupted the index files. Although depending on how exactly restic accesses the cloud storage, there could be other parts of the setup that cause the corruption.
1 Like
I have read about restic issues on Hetzner from other places, and I will think about switching to another provider if it will happen on a regular basis. My setup is simple, using Rclone as a backend with SFTP.
Rclone config:
[box]
type = sftp
host = uXXXXX.your-storagebox.de
user = uXXXXX
port = 23
pass = <obscured-password>
The sftp backend of restic shouldn’t be prone to store damaged files in the repository. restic 0.17.0 should also solve the sftp performance problems.
It could be the hardware of Hetzner. I don’t know how reliable is that provider!