Fatal: load<index/xxxxxxxxx>: invalid data returned

Getting Help
#1

Hi guys. I’m trying to recover some files from an old repo. However, I’m getting the error below. I found Repository Corrupted? Error: load <snapshot/xxxxxxx>: invalid data returned, but that’s specifically focused on a snapshot, not an index.

Can anyone help?

Thanks in advance!

$ restic -r sftp:root@172.16.5.5:/mnt/Backups/gulf2 check
using temporary cache in /tmp/restic-check-cache-630392702
enter password for repository:
repository 8fb8e361 opened successfully, password is correct
created new cache in /tmp/restic-check-cache-630392702
create exclusive lock for repository
load indexes
pack d8ce1de6 contained in several indexes: {0f8b35f2 ee080a5d}
pack 5859804f contained in several indexes: {02f363ac 066cdd94}
pack 6ffc3d1a contained in several indexes: {0f8b35f2 9b0628d5}
This is non-critical, you can run `restic rebuild-index' to correct this
check all packs
check snapshots, trees and blobs
no errors were found snapshots
[3:31] 100.00%  2 / 2 snapshots
$ restic -r sftp:root@172.16.5.5:/mnt/Backups/gulf2 rebuild-index
enter password for repository:
repository 8fb8e361 opened successfully, password is correct
loading indexes...
Fatal: load <index/541511509c>: invalid data returned
#2

Hi there!

For starters, which restic version are you using? If not 0.12.0 or newer, please re-run the rebuild-index command with the most recent version so we know the problem is reproduced with that.

#3

Sorry about that…my post was definitely lacking detail.

Version: restic 0.12.0 compiled with go1.15.8 on linux/amd64

I already tried running rebuild-index before I posted. This is what I get:

$ restic -r sftp:root@172.16.5.5:/mnt/Backups/gulf2 rebuild-index
enter password for repository:
repository 8fb8e361 opened successfully, password is correct
loading indexes...
Fatal: load <index/bb4c685762>: invalid data returned
#4

This error means that the data restic loaded from the sftp server has been modified. The file names in the repo always correspond to the SHA256 hash of its contents. When restic accesses a complete file, e.g. when loading an index, it first checks that the hash matches the filename. In this case the hash does not match, so it prints an error and aborts. The ID you see there (bb4c685762) is the beginning of the SHA256 hash. In our experience, this usually indicates a hardware problem, either the memory (of the server or the machine running restic) or, more likely, the storage medium.

Can you run sha256sum on the file index/bb4c685762* on the server? Or download the file to the local machine and run it there?

In general, the index files are just an optimization and restic is able to rebuild an index from scratch just by looking at the data files. In my opinion, the first priority for you should be to find out where the error comes from, before proceeding to repair the repo.

#5

Ok, thank you. At present, I have a backup of this data so it’s not catastrophic (yet).

The hardware in question is a Freenas Mini. It’s old (5-ish years) but does have ECC ram.

Here’s the sha256sum:

# sha256 bb4c68*

SHA256 (bb4c685762781316b632b7671a2731224c2dd46476a4d4cffb492005e117a221) = bb4c685762781316b632b7671a2731224c2dd46476a4d4cffb492005e117a221

#6

What is the kernel version on the host running restic as well as the sftp server?

1 Like
#7 
FreeBSD freenas 11.3-RELEASE-p6 FreeBSD 11.3-RELEASE-p6 #0 r325575+d5b100edfcb(HEAD): Fri Feb 21 18:53:26 UTC 2020     root@tnbuild02.tn.ixsystems.com:/freenas-releng/freenas/_BE/objs/freenas-releng/freenas/_BE/os/sys/FreeNAS.amd64  amd64

SSH-2.0-OpenSSH_8.0-hpn14v15

I may be worth noting that I have other restic repositories on this server and they’re working fine…it’s just this on in particular which is having issues.

Thanks!

#8

The kernel version on the client where I’m running restic is:

Linux deepdark 5.4.0-65-generic #73-Ubuntu SMP Mon Jan 18 17:25:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
#9

Could be a kernel bug, try upgrading to 5.4.2.

2 Likes
#10

@cdhowie hey…you appear to be on to something. I tried a check from my Macbook and it worked!

Since this appears to be a Linux bug, shall I assume you need no issue to be created?

Thanks so much for the help!

restic -r sftp:root@172.16.5.5:/mnt/Backups/gulf2 check
using temporary cache in /var/folders/w2/t5x94yvn4f1c6cr9fy081z140000gn/T/restic-check-cache-506008388
repository 8fb8e361 opened successfully, password is correct
created new cache in /var/folders/w2/t5x94yvn4f1c6cr9fy081z140000gn/T/restic-check-cache-506008388
create exclusive lock for repository
load indexes
pack d8ce1de6 contained in several indexes: {0f8b35f2 ee080a5d}
pack 6ffc3d1a contained in several indexes: {0f8b35f2 9b0628d5}
pack 5859804f contained in several indexes: {02f363ac 066cdd94}
This is non-critical, you can run `restic rebuild-index' to correct this
check all packs
check snapshots, trees and blobs
[4:42] 50.00%  1 / 2 snapshots
[4:43] 50.00%  1 / 2 snapshots
[4:43] 50.00%  1 / 2 snapshots
[4:43] 50.00%  1 / 2 snapshots
no errors were found snapshots
[5:50] 100.00%  2 / 2 snapshots
#11

Yep, I don’t think we need a bug report. It’s not a restic issue, it’s likely that kernel issue which should be corrected when you upgrade to 5.4.2+. Definitely let us know what happens after you upgrade. Confirmation that it resolved the issue would be appreciated so we can mark the thread resolved.

#12

Will do. I may not be able to upgrade for a bit, but I’ll update this thread when I do. Thanks!

#13

Glad you found the underlying issue!

Just so that you aware of it: the kernel you’re running can cause data loss. What you saw was just a bogus computation of a SHA256 hash, but it can be much more severe. The kernel causes memory corruption: Go issue, Kernel Bugtracker

It’s much more likely that Go programs are affected due to the way the Go runtime works, but it also happens with other programs. If I were you I’d try to upgrade as soon as possible, or (if that’s not possible) use a different machine for now. :slight_smile:

1 Like