Restic slice bounds out of range?

Getting Help
1

Hi,
I was wondering if someone else has this issue before?
Currently running restic - version
restic 0.9.5 compiled with go1.12.4 on linux/amd64

using rescript to run restic
currently have the same config for 6 other servers and this one seem to be putting problems was working great before

Thank you

scan finished in 8542.755s: 1459090 files, 696.617 GiB
panic: runtime error: slice bounds out of range

goroutine 93 [running]:
github.com/restic/chunker.(*Chunker).Next(0xc00017b900, 0xc0317f4000, 0x0, 0x800   000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /restic/vendor/github.com/restic/chunker/chunker.go:292 +0xcb4
github.com/restic/restic/internal/archiver.(*FileSaver).saveFile(0xc000389400, 0   xf84780, 0xc0144634c0, 0xc00017b900, 0xc017603da0, 0x5d, 0xf8f6c0, 0xc00000ed90,    0xf8aa40, 0xc00fd15ee0, ...)
        /restic/internal/archiver/file_saver.go:176 +0x3cc
github.com/restic/restic/internal/archiver.(*FileSaver).worker(0xc000389400, 0xf   84780, 0xc0144634c0, 0xc01a106fc0)
        /restic/internal/archiver/file_saver.go:244 +0x2e6
github.com/restic/restic/internal/archiver.NewFileSaver.func2(0x0, 0x0)
        /restic/internal/archiver/file_saver.go:92 +0x7a
gopkg.in/tomb%2ev2.(*Tomb).run(0xc0003893b0, 0xc01a189ce0)
        /restic/vendor/gopkg.in/tomb.v2/tomb.go:163 +0x2b
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        /restic/vendor/gopkg.in/tomb.v2/tomb.go:159 +0xb7

WARNING!
[backup] failed; exit code 2
2

Judging from a quick search in the forum and the github bugtracker you’re the first to report that crash.

Is that crash reproducible? If yes, you could run the backup in verbose mode restic backup -vv to find out which file is causing the crash? For debugging it would be helpful to use a restic binary built with go >= 1.13 which would also print which slice boundaries were used.

3

Thanks for the reply, so i reran manually but this time no error im going to wait until the cronjob does it automatic and see if it happens again ill post back

4

My guess would be that the error won’t show up again. I took a look at the chunker code and the bounds handling looked correct to me.

Does the cron job backup anything special or just files from a local filesystem?

5

thanks for the reply, it was very odd the conjob run the script of rescript

6

@fd0 and me had a really close look at the chunker function, but we couldn’t find anything pointing towards a bug in the chunker code.
The crash rather looks like a bit-flip in memory, so you might want to run a memcheck.

Which type of filesystem do you backup?

2 Likes
7

Thanks for the reply, @killmasta93 so what i did find out was that again i was getting that error, when i checked the logs restic was running at the same time as the vz dumps (running proxmox) which causes i think a panic but the dump was running the same VM which is backing up the files, i turned it off to see and it worked flawless but thanks again

1 Like
8

@farzadha2 thanks i did pm you because you were also running rescript thinking it was the script itself but going to try that and let you know

9

I’m not sure whether I understood you correctly, do you also get the slice bounds out of range like @killmasta93?

10

yes correct i was also getting the same issue as he was until i found that was the issue

1 Like
11

Which version of restic are you using? If it’s restic v0.9.6 I’d be interested in the full error output.

12

using the same restic version as @killmasta93 0.9.6 im going to update it to see what happens ill post back again thank you

1 Like
13

It would be very helpful for us if you can reproduce the issue. Please let us know if that’s the case :slight_smile:

14

Thanks for the replies, @farzadha2 i changed the vz dumps to do later after restic does the backup and so far so good @fd0 i will postback this week if i get the error again thank you again for everything

1 Like
15

so very odd again yesterday i got the error so i ran restic backup -vv and did not show any errors but today i got this error

scan finished in 1743.505s: 1460361 files, 698.083 GiB
uploaded intermediate index 13a9d1f5
uploaded intermediate index c7b410c6
panic: runtime error: slice bounds out of range [:4294967283] with capacity 524288

goroutine 100 [running]:
github.com/restic/chunker.(*Chunker).Next(0xc014dec000, 0xc047fea000, 0x0, 0x800000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/restic/vendor/github.com/restic/chunker/chunker.go:292 +0xc89
github.com/restic/restic/internal/archiver.(*FileSaver).saveFile(0xc000250eb0, 0xf20fe0, 0xc010154e40, 0xc014dec000, 0xc0462a5130, 0x4a, 0xf2b6a0, 0xc0102ff978, 0xf26e40, 0xc006692340, ...)
	/restic/internal/archiver/file_saver.go:176 +0x39e
github.com/restic/restic/internal/archiver.(*FileSaver).worker(0xc000250eb0, 0xf20fe0, 0xc010154e40, 0xc014b433e0)
	/restic/internal/archiver/file_saver.go:244 +0x2e5
github.com/restic/restic/internal/archiver.NewFileSaver.func2(0x0, 0x0)
	/restic/internal/archiver/file_saver.go:92 +0x7a
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000250e60, 0xc014c0bcb0)
	/restic/vendor/gopkg.in/tomb.v2/tomb.go:163 +0x2b
created by gopkg.in/tomb%2ev2.(*Tomb).Go
	/restic/vendor/gopkg.in/tomb.v2/tomb.go:159 +0xc7

WARNING!
[backup] failed; exit code 2
--------------------------------------------------------------------------------
End: Mon Apr 13 2020 05:32:16 PM
Duration: 1 hour, 22 minutes and 14 seconds
================================================================================
                                  BACKUP ENDED
================================================================================
16

Definitely something funky going on here. This number in hex is FFFFFFF3, which looks like integer underflow…

17

The error looks like io.ReadFull(c.rd, buf[:]) returned a negative length (at least that’s the only sensible explanation I see how that slice index got that large). When interpreted as a negative number the length would be -13 (restic casts the length to an uint). Problem is that ReadFull should never return any length below zero. Maybe that’s happening nonetheless :frowning: .

@killmasta93 Which type of filesystem do you backup? Taking half an hour to list 1.5 million files looks rather slow to me. Do you know/have an idea which file causes restic to crash?

1 Like
18

Thanks for the reply, currently backing up file shares on a window server which i mount it on CIFS on proxmox and the NAS so i can run restic on proxmox to backup the windows server files. i think whats odd is that on a previous post at first its really fast then it gets slow as you mention. But if i run restic manually i get no issue its sometimes this issue when i run using rescript so odd

19

Thank you very much! Something odd is going on here, I agree with @MichaelEischer that apparently io.ReadFull() returned a negative number of bytes. That should not happen.

Can you maybe apply the following patch and see if it appears again?

The only explanation I currently have is that the file system you’re reading data from returns odd values. io.ReadFull() is a thin wrapper around the read syscall, which according to the man page does not return negative values other than -1 to signal an error…

1 Like
20

Thank you for the reply, not sure how can i apply the patch? My guess is
download the patch and run the following command? do i need to reboot?

go chunker.patch

Thank you