Fatal: DecodeIndex: invalid character '\x02' in string literal from one machine

Getting Help
1

I have a SFTP repository that works from one machine without issues, but fails with an error from another , new, machine trying to use the same repository.

The error when trying to do the initial backup is

repository 725a1d24 opened (version 2, compression level auto)
no parent snapshot found, will read all files
Fatal: DecodeIndex: invalid character 'd' after object key:value pair

The machine where it is working:
freebsd-version
13.0-RELEASE-p7

restic version
restic 0.15.2 compiled with go1.20.3 on freebsd/amd64

The machine where it is not working
freebsd-version
13.1-RELEASE-p5

restic version
restic 0.15.2 compiled with go1.20.3 on freebsd/amd64

Aren’t both machines reading the same index files from the SFTP repo?

Also, doing a ls against the same repo works from the existing machine and crashes from the new machine.

Crash for ls from new machine:

blob 4f85841d7bdf8ebbfc924020cccfc0a23f219770350041ad6f2f9fb9655cef28 returned invalid hash
github.com/restic/restic/internal/repository.(*Repository).LoadBlob
        github.com/restic/restic/internal/repository/repository.go:330
github.com/restic/restic/internal/restic.LoadTree
        github.com/restic/restic/internal/restic/tree.go:113
github.com/restic/restic/internal/walker.walk
        github.com/restic/restic/internal/walker/walker.go:102
github.com/restic/restic/internal/walker.walk
        github.com/restic/restic/internal/walker/walker.go:122
github.com/restic/restic/internal/walker.walk
        github.com/restic/restic/internal/walker/walker.go:122
github.com/restic/restic/internal/walker.walk
        github.com/restic/restic/internal/walker/walker.go:122
github.com/restic/restic/internal/walker.walk
        github.com/restic/restic/internal/walker/walker.go:122
github.com/restic/restic/internal/walker.walk
        github.com/restic/restic/internal/walker/walker.go:122
github.com/restic/restic/internal/walker.Walk
        github.com/restic/restic/internal/walker/walker.go:51
main.runLs
        github.com/restic/restic/cmd/restic/cmd_ls.go:224
main.glob..func15
        github.com/restic/restic/cmd/restic/cmd_ls.go:45
github.com/spf13/cobra.(*Command).execute
        github.com/spf13/cobra@v1.6.1/command.go:916
github.com/spf13/cobra.(*Command).ExecuteC
        github.com/spf13/cobra@v1.6.1/command.go:1044
github.com/spf13/cobra.(*Command).Execute
        github.com/spf13/cobra@v1.6.1/command.go:968
github.com/spf13/cobra.(*Command).ExecuteContext
        github.com/spf13/cobra@v1.6.1/command.go:961
main.main
        github.com/restic/restic/cmd/restic/main.go:96
runtime.main
        runtime/proc.go:250
runtime.goexit
        runtime/asm_amd64.s:1598

From what I am seeing seems there is something different, or missing, in the new that is causing the crashes.

Any pointers or ideas what may be the issue?

2

What is sftp server?

Have you tried to sftp directly to it from machine running freebsd 13.1? and get some blobs?

Worse case it is problem with GitHub - pkg/sftp: SFTP support for the go.crypto/ssh package and this freebsd version.

As a workaround you can try to connect via rclone sftp remote. rclone uses newer version of go sftp package (v1.13.6 vs v1.13.5 in restic)

3

What is sftp server?

The SFTP server is a FreeBSD 13.1 machine.

Have you tried to sftp directly to it from machine running freebsd 13.1? and get some blobs?
Do you mean like trying to scp down some files from the SFTP server?

Tried this and did not have any issues:
scp :/backups/restic/data/ff/ff11395476193b6a817da1574d8f88829e4d7d18e961b9e4b8b0fa69b64e0c7a .

File was downloaded without issues.

As a workaround you can try to connect via rclone sftp remote

Is this what I should be looking at rclone serve restic?

4

so looks like server is ok - freeBSD 13.1 sftp client is ok - problem is in restic sftp client it seems.

I would spin rclone sftp remote - check if rclone can cooperate with your sftp server. Both restic and rclone use the same go library - but rclone has newer one.

If yes then as a workaround I would use restic rclone and maybe report to GitHub - pkg/sftp: SFTP support for the go.crypto/ssh package

5

You do not have to look at this at all:) restic will take care of this.

What you have to do is configure rclone sftp remote,

~/.config/rclone/rclone.config:

[sftpremote]
host = xx.xx.xx.xx
user = sftpuser
port = 22
pass = 
key_file =

check if it works:

rclone lsd sftpremote:

this should list directories in root folder of your sftp server

and then just use in restic:

https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html?highlight=rclone#other-services-via-rclone

e.g.:

restic -r rclone:sftpremote:resticRepoFolder ls latest

effectively instead of restic sftp client you will use rclone one.

6

Lets see if it works.

7

I believe the machine in question has issues, likely networking, since with Rclone got several, different, errors. A few times the listing of ls actually started and then failed in the middle.

Going to test another machine in the same provider, Vultr, and see if the issue also happens. Lastly, another process that runs throughout the day is also showing random disconnects which leads me to believe is not a restic issue.

1 Like
8

Thx for sharing your findings.

9

There are only two possibilities for these errors to occur: either the restic client that created the snapshot encounters memory corruption or the restic client that tries to read the snapshot does.

These errors are completely unrelated to network problems or problems at the repository storage.

10

Thanks for the feedback. Tested another machine with the exact same setup (used ansible playbooks so the machines have the exact same setup) and there are no issues against the same SFTP repo.

So whatever the issue, it seems to be that one machine and not a Restic problem.

1 Like