bump? anyone could shed some light on how to patch the above?
Iām sorry, I was busy. Turns out itās not so easy to patch the chunker with knowing exactly what to do, so Iāve pushed the branch debug-chunker. You can build restic with this patch as follows:
git clone https://github.com/restic/restic
cd restic/cmd/restic
git checkout debug-chunker
go build
1 Like
Thanks for the reply, no worries so i ran the patch but not sure if it ran correctly this is the output
root@prometheus7:~/restic/cmd/restic# git checkout debug-chunker
Already on 'debug-chunker'
Your branch is up to date with 'origin/debug-chunker'.
root@prometheus7:~/restic/cmd/restic# go build
cmd_mount.go:22:2: cannot find package "bazil.org/fuse" in any of:
/usr/local/go/src/bazil.org/fuse (from $GOROOT)
/root/go/src/bazil.org/fuse (from $GOPATH)
cmd_mount.go:23:2: cannot find package "bazil.org/fuse/fs" in any of:
/usr/local/go/src/bazil.org/fuse/fs (from $GOROOT)
/root/go/src/bazil.org/fuse/fs (from $GOPATH)
cmd_backup.go:20:2: cannot find package "github.com/restic/restic/internal/archi ver" in any of:
/usr/local/go/src/github.com/restic/restic/internal/archiver (from $GORO OT)
/root/go/src/github.com/restic/restic/internal/archiver (from $GOPATH)
cmd_cat.go:10:2: cannot find package "github.com/restic/restic/internal/backend" in any of:
/usr/local/go/src/github.com/restic/restic/internal/backend (from $GOROO T)
/root/go/src/github.com/restic/restic/internal/backend (from $GOPATH)
global.go:16:2: cannot find package "github.com/restic/restic/internal/backend/a zure" in any of:
/usr/local/go/src/github.com/restic/restic/internal/backend/azure (from $GOROOT)
/root/go/src/github.com/restic/restic/internal/backend/azure (from $GOPA TH)
global.go:17:2: cannot find package "github.com/restic/restic/internal/backend/b 2" in any of:
/usr/local/go/src/github.com/restic/restic/internal/backend/b2 (from $GO ROOT)
/root/go/src/github.com/restic/restic/internal/backend/b2 (from $GOPATH)
global.go:18:2: cannot find package "github.com/restic/restic/internal/backend/g s" in any of:
/usr/local/go/src/github.com/restic/restic/internal/backend/gs (from $GO ROOT)
/root/go/src/github.com/restic/restic/internal/backend/gs (from $GOPATH)
global.go:19:2: cannot find package "github.com/restic/restic/internal/backend/l ocal" in any of:
/usr/local/go/src/github.com/restic/restic/internal/backend/local (from $GOROOT)
/root/go/src/github.com/restic/restic/internal/backend/local (from $GOPA TH)
global.go:20:2: cannot find package "github.com/restic/restic/internal/backend/l ocation" in any of:
/usr/local/go/src/github.com/restic/restic/internal/backend/location (fr om $GOROOT)
/root/go/src/github.com/restic/restic/internal/backend/location (from $G OPATH)
global.go:21:2: cannot find package "github.com/restic/restic/internal/backend/r clone" in any of:
/usr/local/go/src/github.com/restic/restic/internal/backend/rclone (from $GOROOT)
/root/go/src/github.com/restic/restic/internal/backend/rclone (from $GOP ATH)
global.go:22:2: cannot find package "github.com/restic/restic/internal/backend/r est" in any of:
/usr/local/go/src/github.com/restic/restic/internal/backend/rest (from $ GOROOT)
/root/go/src/github.com/restic/restic/internal/backend/rest (from $GOPAT H)
global.go:23:2: cannot find package "github.com/restic/restic/internal/backend/s 3" in any of:
/usr/local/go/src/github.com/restic/restic/internal/backend/s3 (from $GO ROOT)
/root/go/src/github.com/restic/restic/internal/backend/s3 (from $GOPATH)
global.go:24:2: cannot find package "github.com/restic/restic/internal/backend/s ftp" in any of:
/usr/local/go/src/github.com/restic/restic/internal/backend/sftp (from $ GOROOT)
/root/go/src/github.com/restic/restic/internal/backend/sftp (from $GOPAT H)
global.go:25:2: cannot find package "github.com/restic/restic/internal/backend/s wift" in any of:
/usr/local/go/src/github.com/restic/restic/internal/backend/swift (from $GOROOT)
/root/go/src/github.com/restic/restic/internal/backend/swift (from $GOPA TH)
cmd_cache.go:10:2: cannot find package "github.com/restic/restic/internal/cache" in any of:
/usr/local/go/src/github.com/restic/restic/internal/cache (from $GOROOT)
/root/go/src/github.com/restic/restic/internal/cache (from $GOPATH)
cmd_check.go:13:2: cannot find package "github.com/restic/restic/internal/checke r" in any of:
/usr/local/go/src/github.com/restic/restic/internal/checker (from $GOROO T)
/root/go/src/github.com/restic/restic/internal/checker (from $GOPATH)
cleanup.go:10:2: cannot find package "github.com/restic/restic/internal/debug" i n any of:
/usr/local/go/src/github.com/restic/restic/internal/debug (from $GOROOT)
/root/go/src/github.com/restic/restic/internal/debug (from $GOPATH)
cmd_backup.go:22:2: cannot find package "github.com/restic/restic/internal/error s" in any of:
/usr/local/go/src/github.com/restic/restic/internal/errors (from $GOROOT )
/root/go/src/github.com/restic/restic/internal/errors (from $GOPATH)
cmd_find.go:13:2: cannot find package "github.com/restic/restic/internal/filter" in any of:
/usr/local/go/src/github.com/restic/restic/internal/filter (from $GOROOT )
/root/go/src/github.com/restic/restic/internal/filter (from $GOPATH)
cmd_backup.go:23:2: cannot find package "github.com/restic/restic/internal/fs" i n any of:
/usr/local/go/src/github.com/restic/restic/internal/fs (from $GOROOT)
/root/go/src/github.com/restic/restic/internal/fs (from $GOPATH)
cmd_mount.go:20:2: cannot find package "github.com/restic/restic/internal/fuse" in any of:
/usr/local/go/src/github.com/restic/restic/internal/fuse (from $GOROOT)
/root/go/src/github.com/restic/restic/internal/fuse (from $GOPATH)
cmd_list.go:7:2: cannot find package "github.com/restic/restic/internal/index" i n any of:
/usr/local/go/src/github.com/restic/restic/internal/index (from $GOROOT)
/root/go/src/github.com/restic/restic/internal/index (from $GOPATH)
global.go:29:2: cannot find package "github.com/restic/restic/internal/limiter" in any of:
/usr/local/go/src/github.com/restic/restic/internal/limiter (from $GOROO T)
/root/go/src/github.com/restic/restic/internal/limiter (from $GOPATH)
cmd_migrate.go:4:2: cannot find package "github.com/restic/restic/internal/migra tions" in any of:
/usr/local/go/src/github.com/restic/restic/internal/migrations (from $GO ROOT)
/root/go/src/github.com/restic/restic/internal/migrations (from $GOPATH)
cmd_options.go:6:2: cannot find package "github.com/restic/restic/internal/optio ns" in any of:
/usr/local/go/src/github.com/restic/restic/internal/options (from $GOROO T)
/root/go/src/github.com/restic/restic/internal/options (from $GOPATH)
cmd_backup.go:24:2: cannot find package "github.com/restic/restic/internal/repos itory" in any of:
/usr/local/go/src/github.com/restic/restic/internal/repository (from $GO ROOT)
/root/go/src/github.com/restic/restic/internal/repository (from $GOPATH)
cmd_backup.go:25:2: cannot find package "github.com/restic/restic/internal/resti c" in any of:
/usr/local/go/src/github.com/restic/restic/internal/restic (from $GOROOT )
/root/go/src/github.com/restic/restic/internal/restic (from $GOPATH)
cmd_restore.go:10:2: cannot find package "github.com/restic/restic/internal/rest orer" in any of:
/usr/local/go/src/github.com/restic/restic/internal/restorer (from $GORO OT)
/root/go/src/github.com/restic/restic/internal/restorer (from $GOPATH)
cmd_self_update.go:9:2: cannot find package "github.com/restic/restic/internal/s elfupdate" in any of:
/usr/local/go/src/github.com/restic/restic/internal/selfupdate (from $GO ROOT)
/root/go/src/github.com/restic/restic/internal/selfupdate (from $GOPATH)
cmd_backup.go:26:2: cannot find package "github.com/restic/restic/internal/textf ile" in any of:
/usr/local/go/src/github.com/restic/restic/internal/textfile (from $GORO OT)
/root/go/src/github.com/restic/restic/internal/textfile (from $GOPATH)
cmd_backup.go:27:2: cannot find package "github.com/restic/restic/internal/ui" i n any of:
/usr/local/go/src/github.com/restic/restic/internal/ui (from $GOROOT)
/root/go/src/github.com/restic/restic/internal/ui (from $GOPATH)
cmd_backup.go:28:2: cannot find package "github.com/restic/restic/internal/ui/js on" in any of:
/usr/local/go/src/github.com/restic/restic/internal/ui/json (from $GOROO T)
/root/go/src/github.com/restic/restic/internal/ui/json (from $GOPATH)
cmd_cache.go:13:2: cannot find package "github.com/restic/restic/internal/ui/tab le" in any of:
/usr/local/go/src/github.com/restic/restic/internal/ui/table (from $GORO OT)
/root/go/src/github.com/restic/restic/internal/ui/table (from $GOPATH)
cmd_backup.go:29:2: cannot find package "github.com/restic/restic/internal/ui/te rmstatus" in any of:
/usr/local/go/src/github.com/restic/restic/internal/ui/termstatus (from $GOROOT)
/root/go/src/github.com/restic/restic/internal/ui/termstatus (from $GOPA TH)
cmd_dump.go:16:2: cannot find package "github.com/restic/restic/internal/walker" in any of:
/usr/local/go/src/github.com/restic/restic/internal/walker (from $GOROOT )
/root/go/src/github.com/restic/restic/internal/walker (from $GOPATH)
cmd_backup.go:17:2: cannot find package "github.com/spf13/cobra" in any of:
/usr/local/go/src/github.com/spf13/cobra (from $GOROOT)
/root/go/src/github.com/spf13/cobra (from $GOPATH)
cmd_generate.go:8:2: cannot find package "github.com/spf13/cobra/doc" in any of:
/usr/local/go/src/github.com/spf13/cobra/doc (from $GOROOT)
/root/go/src/github.com/spf13/cobra/doc (from $GOPATH)
global.go:39:2: cannot find package "golang.org/x/crypto/ssh/terminal" in any of :
/usr/local/go/src/golang.org/x/crypto/ssh/terminal (from $GOROOT)
/root/go/src/golang.org/x/crypto/ssh/terminal (from $GOPATH)
cmd_backup.go:18:2: cannot find package "gopkg.in/tomb.v2" in any of:
/usr/local/go/src/gopkg.in/tomb.v2 (from $GOROOT)
/root/go/src/gopkg.in/tomb.v2 (from $GOPATH)
Which version of go do you use (go version)?
Thanks for the reply
root@prometheus7:~/restic/cmd/restic# go version
go version go1.10.1 linux/amd64Hmm, the debug branch requires at least go 1.11. Go 1.13 or 1.14 would be even better.
1 Like
Thank you so much so i did the following i think it patched it
root@prometheus7:~# go version
go version go1.14.2 linux/amd64
root@prometheus7:~# cd restic/cmd/restic
root@prometheus7:~/restic/cmd/restic# git checkout debug-chunker
Already on 'debug-chunker'
Your branch is up to date with 'origin/debug-chunker'.
root@prometheus7:~/restic/cmd/restic# go build
go: downloading github.com/spf13/cobra v0.0.3
go: downloading bazil.org/fuse v0.0.0-20191225072544-27e78e7d88df
go: downloading github.com/pkg/errors v0.8.1
go: downloading github.com/pkg/sftp v1.10.0
go: downloading google.golang.org/api v0.3.2
go: downloading gopkg.in/tomb.v2 v2.0.0-20161208151619-d5d1b5820637
go: downloading golang.org/x/oauth2 v0.0.0-20190402181905-9f3314589c9a
go: downloading golang.org/x/crypto v0.0.0-20190513172903-22d7a77e9e5f
go: downloading github.com/minio/minio-go/v6 v6.0.43
go: downloading cloud.google.com/go v0.37.4
go: downloading golang.org/x/net v0.0.0-20190522155817-f3200d17e092
go: downloading golang.org/x/sync v0.0.0-20190423024810-112230192c58
go: downloading github.com/kr/fs v0.1.0
go: downloading github.com/cpuguy83/go-md2man v1.0.10
go: downloading github.com/russross/blackfriday v1.5.2
go: downloading github.com/spf13/pflag v1.0.3
go: downloading github.com/cespare/xxhash v1.1.0
go: downloading github.com/pkg/xattr v0.4.1
go: downloading github.com/ncw/swift v1.0.47
go: downloading golang.org/x/sys v0.0.0-20191210023423-ac6580df4449
go: downloading github.com/cenkalti/backoff v2.1.1+incompatible
go: downloading golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2
go: downloading github.com/Azure/azure-sdk-for-go v27.3.0+incompatible
go: downloading github.com/kurin/blazer v0.5.3
go: downloading gopkg.in/yaml.v2 v2.2.2
go: downloading github.com/mitchellh/go-homedir v1.1.0
go: downloading github.com/elithrar/simple-scrypt v1.3.0
go: downloading gopkg.in/ini.v1 v1.42.0
go: downloading github.com/minio/sha256-simd v0.1.1
go: downloading github.com/juju/ratelimit v1.0.1
go: downloading go.opencensus.io v0.20.2
go: downloading google.golang.org/grpc v1.20.1
go: downloading google.golang.org/genproto v0.0.0-20190418145605-e7d98fc518a7
go: downloading github.com/golang/protobuf v1.3.1
go: downloading github.com/hashicorp/golang-lru v0.5.1
go: downloading github.com/marstr/guid v1.1.0
go: downloading github.com/Azure/go-autorest/autorest v0.9.2
go: downloading github.com/satori/go.uuid v1.2.0
go: downloading github.com/Azure/go-autorest/tracing v0.5.0
go: downloading github.com/Azure/go-autorest/logger v0.1.0
go: downloading github.com/Azure/go-autorest/autorest/adal v0.5.0
go: downloading github.com/dgrijalva/jwt-go v3.2.0+incompatible
go: downloading github.com/Azure/go-autorest/autorest/date v0.1.0
1 Like
The output looks like it built the patched version. I wonder what the debug code will output when the chunker error shows up again.
ill postback tonight how it went
2 Likes
Hmm, the patch is only supposed to print an additional warning if the file-reading in the chunker encounters a negative amount of bytes (sound strange and it is). The only change that could be a fix for the crash youāve observed is the change to Go 1.14 (and maybe the newer restic branch, but I donāt recall any changes that could have fixed that crash).
very odd again today got the issue
panic: runtime error: slice bounds out of range [:4294967283] with capacity 524288
goroutine 101 [running]:
github.com/restic/chunker.(*Chunker).Next(0xc014654000, 0xc0208a4000, 0x0, 0x800000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/restic/vendor/github.com/restic/chunker/chunker.go:292 +0xc89
github.com/restic/restic/internal/archiver.(*FileSaver).saveFile(0xc014025090, 0xf20fe0, 0xc0112f2840, 0xc014654000, 0xc01af30c80, 0x4a, 0xf2b6a0, 0xc00012fad8, 0xf26e40, 0xc009091930, ...)
/restic/internal/archiver/file_saver.go:176 +0x39e
github.com/restic/restic/internal/archiver.(*FileSaver).worker(0xc014025090, 0xf20fe0, 0xc0112f2840, 0xc011bf5b60)
/restic/internal/archiver/file_saver.go:244 +0x2e5
github.com/restic/restic/internal/archiver.NewFileSaver.func2(0x0, 0x0)
/restic/internal/archiver/file_saver.go:92 +0x7a
gopkg.in/tomb%2ev2.(*Tomb).run(0xc014025040, 0xc0140659b0)
/restic/vendor/gopkg.in/tomb.v2/tomb.go:163 +0x2b
created by gopkg.in/tomb%2ev2.(*Tomb).Go
/restic/vendor/gopkg.in/tomb.v2/tomb.go:159 +0xc7Thatās not quite correct, the patch should return an error from the chunker if a negative number of bytes is encountered.
@killmasta93 Can you please make sure that the binary is the patched one? E.g. run strings on it an grep for ReadFull returned negative number of bytesā¦
This path in the stack trace makes me think that maybe youāre not running the patched binary:
/restic/vendor/github.com/restic/chunker/chunker.go:292 +0xc89
It should have been something like this:
/restic/chunker/chunker.go
Thanks for the reply, im somewhat confused so i did not patch it?
This is what i got
root@prometheus7:~/restic/chunker# strings chunker.go
package chunker
import (
"errors"
"fmt"
"io"
"sync"
const (
kiB = 1024
miB = 1024 * kiB
// WindowSize is the size of the sliding window.
windowSize = 64
// MinSize is the default minimal size of a chunk.
MinSize = 512 * kiB
// MaxSize is the default maximal size of a chunk.
MaxSize = 8 * miB
chunkerBufSize = 512 * kiB
type tables struct {
out [256]Pol
mod [256]Pol
// cache precomputed tables, these are read-only anyway
var cache struct {
entries map[Pol]tables
sync.Mutex
func init() {
cache.entries = make(map[Pol]tables)
// Chunk is one content-dependent chunk of bytes whose end was cut when the
// Rabin Fingerprint had the value stored in Cut.
type Chunk struct {
Start uint
Length uint
Cut uint64
Data []byte
type chunkerState struct {
window [windowSize]byte
wpos uint
buf []byte
bpos uint
bmax uint
start uint
count uint
pos uint
pre uint // wait for this many bytes before start calculating an new chunk
digest uint64
type chunkerConfig struct {
MinSize, MaxSize uint
pol Pol
polShift uint
tables tables
tablesInitialized bool
splitmask uint64
rd io.Reader
closed bool
// Chunker splits content with Rabin Fingerprints.
type Chunker struct {
chunkerConfig
chunkerState
// SetAverageBits allows to control the frequency of chunk discovery:
// the lower averageBits, the higher amount of chunks will be identified.
// The default value is 20 bits, so chunks will be of 1MiB size on average.
func (c *Chunker) SetAverageBits(averageBits int) {
c.splitmask = (1 << uint64(averageBits)) - 1
// New returns a new Chunker based on polynomial p that reads from rd.
func New(rd io.Reader, pol Pol) *Chunker {
return NewWithBoundaries(rd, pol, MinSize, MaxSize)
// NewWithBoundaries returns a new Chunker based on polynomial p that reads from
// rd and custom min and max size boundaries.
func NewWithBoundaries(rd io.Reader, pol Pol, min, max uint) *Chunker {
c := &Chunker{
chunkerState: chunkerState{
buf: make([]byte, chunkerBufSize),
},
chunkerConfig: chunkerConfig{
pol: pol,
rd: rd,
MinSize: min,
MaxSize: max,
splitmask: (1 << 20) - 1, // aim to create chunks of 20 bits or about 1MiB on average.
},
c.reset()
return c
// Reset reinitializes the chunker with a new reader and polynomial.
func (c *Chunker) Reset(rd io.Reader, pol Pol) {
c.ResetWithBoundaries(rd, pol, MinSize, MaxSize)
// ResetWithBoundaries reinitializes the chunker with a new reader, polynomial
// and custom min and max size boundaries.
func (c *Chunker) ResetWithBoundaries(rd io.Reader, pol Pol, min, max uint) {
*c = Chunker{
chunkerState: chunkerState{
buf: c.buf,
},
chunkerConfig: chunkerConfig{
pol: pol,
rd: rd,
MinSize: min,
MaxSize: max,
splitmask: (1 << 20) - 1,
},
c.reset()
func (c *Chunker) reset() {
c.polShift = uint(c.pol.Deg() - 8)
c.fillTables()
for i := 0; i < windowSize; i++ {
c.window[i] = 0
c.closed = false
c.digest = 0
c.wpos = 0
c.count = 0
c.digest = c.slide(c.digest, 1)
c.start = c.pos
// do not start a new chunk unless at least MinSize bytes have been read
c.pre = c.MinSize - windowSize
// fillTables calculates out_table and mod_table for optimization. This
// implementation uses a cache in the global variable cache.
func (c *Chunker) fillTables() {
// if polynomial hasn't been specified, do not compute anything for now
if c.pol == 0 {
return
c.tablesInitialized = true
// test if the tables are cached for this polynomial
cache.Lock()
defer cache.Unlock()
if t, ok := cache.entries[c.pol]; ok {
c.tables = t
return
// calculate table for sliding out bytes. The byte to slide out is used as
// the index for the table, the value contains the following:
// out_table[b] = Hash(b || 0 || ... || 0)
// \ windowsize-1 zero bytes /
// To slide out byte b_0 for window size w with known hash
// H := H(b_0 || ... || b_w), it is sufficient to add out_table[b_0]:
// H(b_0 || ... || b_w) + H(b_0 || 0 || ... || 0)
// = H(b_0 + b_0 || b_1 + 0 || ... || b_w + 0)
// = H( 0 || b_1 || ... || b_w)
// Afterwards a new byte can be shifted in.
for b := 0; b < 256; b++ {
var h Pol
h = appendByte(h, byte(b), c.pol)
for i := 0; i < windowSize-1; i++ {
h = appendByte(h, 0, c.pol)
c.tables.out[b] = h
// calculate table for reduction mod Polynomial
k := c.pol.Deg()
for b := 0; b < 256; b++ {
// mod_table[b] = A | B, where A = (b(x) * x^k mod pol) and B = b(x) * x^k
//
// The 8 bits above deg(Polynomial) determine what happens next and so
// these bits are used as a lookup to this table. The value is split in
// two parts: Part A contains the result of the modulus operation, part
// B is used to cancel out the 8 top bits so that one XOR operation is
// enough to reduce modulo Polynomial
c.tables.mod[b] = Pol(uint64(b)<<uint(k)).Mod(c.pol) | (Pol(b) << uint(k))
cache.entries[c.pol] = c.tables
// Next returns the position and length of the next chunk of data. If an error
// occurs while reading, the error is returned. Afterwards, the state of the
// current chunk is undefined. When the last chunk has been returned, all
// subsequent calls yield an io.EOF error.
func (c *Chunker) Next(data []byte) (Chunk, error) {
data = data[:0]
if !c.tablesInitialized {
return Chunk{}, errors.New("tables for polynomial computation not initialized")
tabout := c.tables.out
tabmod := c.tables.mod
polShift := c.polShift
// go guarantees the expected behavior for bit shifts even for shift counts
// larger than the value width. Bounding the value of polShift allows the compiler
// to optimize the code for 'digest >> polShift'
if polShift > 53-8 {
return Chunk{}, errors.New("the polynomial must have a degree less than or equal 53")
minSize := c.MinSize
maxSize := c.MaxSize
buf := c.buf
for {
if c.bpos >= c.bmax {
n, err := io.ReadFull(c.rd, buf[:])
if err == io.ErrUnexpectedEOF {
err = nil
}
// io.ReadFull only returns io.EOF when no bytes could be read. If
// this is the case and we're in this branch, there are no more
// bytes to buffer, so this was the last chunk. If a different
// error has occurred, return that error and abandon the current
// chunk.
if err == io.EOF && !c.closed {
c.closed = true
// return current chunk, if any bytes have been processed
if c.count > 0 {
return Chunk{
Start: c.start,
Length: c.count,
Cut: c.digest,
Data: data,
}, nil
}
}
if err != nil {
return Chunk{}, err
}
if n < 0 {
return Chunk{}, fmt.Errorf("ReadFull returned negative number of bytes read: %v", n)
}
c.bpos = 0
c.bmax = uint(n)
// check if bytes have to be dismissed before starting a new chunk
if c.pre > 0 {
n := c.bmax - c.bpos
if c.pre > uint(n) {
c.pre -= uint(n)
data = append(data, buf[c.bpos:c.bmax]...)
c.count += uint(n)
c.pos += uint(n)
c.bpos = c.bmax
continue
}
data = append(data, buf[c.bpos:c.bpos+c.pre]...)
c.bpos += c.pre
c.count += c.pre
c.pos += c.pre
c.pre = 0
add := c.count
digest := c.digest
win := c.window
wpos := c.wpos
for _, b := range buf[c.bpos:c.bmax] {
// slide(b)
// limit wpos before to elide array bound checks
wpos = wpos % windowSize
out := win[wpos]
win[wpos] = b
digest ^= uint64(tabout[out])
wpos++
// updateDigest
index := byte(digest >> polShift)
digest <<= 8
digest |= uint64(b)
digest ^= uint64(tabmod[index])
// end manual inline
add++
if add < minSize {
continue
}
if (digest&c.splitmask) == 0 || add >= maxSize {
i := add - c.count - 1
data = append(data, c.buf[c.bpos:c.bpos+uint(i)+1]...)
c.count = add
c.pos += uint(i) + 1
c.bpos += uint(i) + 1
c.buf = buf
chunk := Chunk{
Start: c.start,
Length: c.count,
Cut: digest,
Data: data,
}
c.reset()
return chunk, nil
}
c.digest = digest
c.window = win
c.wpos = wpos % windowSize
steps := c.bmax - c.bpos
if steps > 0 {
data = append(data, c.buf[c.bpos:c.bpos+steps]...)
c.count += steps
c.pos += steps
c.bpos = c.bmax
func updateDigest(digest uint64, polShift uint, tab tables, b byte) (newDigest uint64) {
index := digest >> polShift
digest <<= 8
digest |= uint64(b)
digest ^= uint64(tab.mod[index])
return digest
func (c *Chunker) slide(digest uint64, b byte) (newDigest uint64) {
out := c.window[c.wpos]
c.window[c.wpos] = b
digest ^= uint64(c.tables.out[out])
c.wpos = (c.wpos + 1) % windowSize
digest = updateDigest(digest, c.polShift, c.tables, b)
return digest
func appendByte(hash Pol, b byte, pol Pol) Pol {
hash <<= 8
hash |= Pol(b)
return hash.Mod(pol)The crashing backup run looks like it used the standard restic 0.9.6 (?) binary and not the one built from the debug-chunker branch. Judging from the build log the new binary should be located in ~/restic/cmd/restic/restic.
thanks for the reply, so what did i do wrong? But didnāt I patch it on the following post?
Your branch is up to date with 'origin/debug-chunker'.
root@prometheus7:~/restic/cmd/restic# go build
Or do i need to call restic from
/restic/cmd/restic/restic
instead of
/usr/bin/rescript
rescript just calls restic and lets the shell resolve it using the PATH environment variable. In other words it probably used the restic binary from /usr/bin or /usr/local/bin, but definitively not the one in /restic/cmd/restic.
You could copy the new restic binary to a separate folder e.g. ~/restic/bin and then add that to the front of the PATH variable by calling: PATH=~/restic/bin:$PATH /usr/bin/rescript. Alternatively you can also call the restic binary directly.
@MichaelEischer thanks for the reply, so what i did copy the restic file and put in the location
root@prometheus7:/media# restic version
restic 0.9.6-dev (compiled manually) compiled with go1.14.2 on linux/amd64
re ran the backup and got this
panic: runtime error: slice bounds out of range [:4294967283] with capacity 524288
goroutine 96 [running]:
github.com/restic/chunker.(*Chunker).Next(0xc015b94000, 0xc01f380000, 0x0, 0x800000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/root/restic/chunker/chunker.go:303 +0xdc0
github.com/restic/restic/internal/archiver.(*FileSaver).saveFile(0xc015f30840, 0xed4640, 0xc00d8f8340, 0xc015b94000, 0xc0222169b0, 0x49, 0xeddb20, 0xc020d14580, 0xed9860, 0xc000871c70, ...)
/root/restic/internal/archiver/file_saver.go:167 +0x39d
github.com/restic/restic/internal/archiver.(*FileSaver).worker(0xc015f30840, 0xed4640, 0xc00d8f8340, 0xc015cd2b40)
/root/restic/internal/archiver/file_saver.go:235 +0x2e5
github.com/restic/restic/internal/archiver.NewFileSaver.func2(0x0, 0x0)
/root/restic/internal/archiver/file_saver.go:88 +0x7a
gopkg.in/tomb%2ev2.(*Tomb).run(0xc0001bd9f0, 0xc015f308a0)
/root/go/pkg/mod/gopkg.in/tomb.v2@v2.0.0-20161208151619-d5d1b5820637/tomb.go:163 +0x38
created by gopkg.in/tomb%2ev2.(*Tomb).Go
/root/go/pkg/mod/gopkg.in/tomb.v2@v2.0.0-20161208151619-d5d1b5820637/tomb.go:159 +0xba
WARNING!
[backup] failed; exit code 2
1 Like
Could you modify
and replace line 269
if n < 0 {
with
if n > cap(buf) {
and test again?
I think that the negative length was a red herring: 0xfffffff3 would be uint32(-13). However, the read length and the slice offset are stored in a 64-bit integer on amd64. Thus it looks like the read syscall reported a read length of 0xfffffff3 (probably after messing up a -13).
Low-level details of what happens to the read length
Iāve followed the internals of io.ReadFull in order to find an explanation for the chunker crash: The file from which the chunker reads, was opened via arch.FS.OpenFile, so weāve got a normal go file object. The loop in ReadFull -> ReadAtLeast can only leave with a negative n if the call to Read did so too. To return a negative n the loop in ReadAtLeast must also terminate early, which is only possible if Read returns an error. The actual read happens in go/internal/poll/fd_unix.go. That FD.Read can either return a non-zero length or an error. The returned length is exactly the length returned by the read syscall. To get the negative length and the early termination would require two calls of the read syscall which must return a negative length first, followed by an error on the second invocation -> that is the only way to get n < 0 is the have err != nil.
Thereās another hurdle to take: The only way to prevent the chunker code from returning with an error, while at the same time having ReadFull return an error is that ReadFull returns ErrUnexpectedEOF. The complication there is that the only way to get that error, is to fulfill n > 0 && err == EOF in ReadAtLeast (Iāve grepped through the go standard library). Remember that earlier on weāve established that n < 0 requires err != nil. Even if err == EOF, then it could not be replaced with ErrUnexpectedEOF as that only happens for n > 0 which conflicts with n < 0. So in summary it is probably impossible to get a negative length.
An alternative would be that the read syscall already returns uint32(-13) = 0xfffffff3. In that case the length would simply be backpropagated to the chunker. On 64-bit platforms Go actually uses 64-bit indices for slices. 0xfffffff3 would be -13 when interpreted as a 32-bit number, but int and uint are 64-bit for amd64 so that doesnāt happen. In case n were negative (e.g. -13), itād have a value of 0xfffffffffffffff3 or 18446744073709551603. The latter was obviously not printed when the chunker crashed.
So to sum up: Thereās only one thing that could have happened and that is that the read syscall reported a read length of 0xfffffff3 (probably after messing up a -13).
2 Likes