BUG: Bad page map in process restic

#1

Hi,

While migrating a lot of backups (I mean I already migrated +100 snapshots without any issue) I faced this error after about 20s (sorry for the long logs).

After rerunning the command it worked like a charm. Should I be worried about it?

My first guess is a memory limit somewhere but I am not sure.

The command I run was

restic backup --verbose --host HOST -r sftp:TARGET:/srv/backup /srv/data

I use:

restic 0.9.4 compiled with go1.11.4 on linux/amd64

Thanks in advance.

In the dmesg:

[Wed Apr  3 17:47:40 2019] BUG: Bad page map in process restic  pte:f000def8f000def8 pmd:00000000
[Wed Apr  3 17:47:40 2019] addr:000000c04d22c000 vm_flags:08100073 anon_vma:ffff8801a6bc2900 mapping:          (null) index:c04d22c
[Wed Apr  3 17:47:40 2019] CPU: 5 PID: 29805 Comm: restic Tainted: G    B         3.13.0-44-generic #73-Ubuntu
[Wed Apr  3 17:47:40 2019] Hardware name: Dell Inc. PowerEdge R210/05KX61, BIOS 1.5.2 10/18/2010
[Wed Apr  3 17:47:40 2019]  ffff880423577500 ffff8801d108fd40 ffffffff81720d86 000000c04d22c000
[Wed Apr  3 17:47:40 2019]  ffff8801d108fd88 ffffffff811757e3 f000def8f000def8 000000000c04d22c
[Wed Apr  3 17:47:40 2019]  000000c04d22c000 ffff880169f71348 ffff880423577500 ffff880422ce3b80
[Wed Apr  3 17:47:40 2019] Call Trace:
[Wed Apr  3 17:47:40 2019]  [<ffffffff81720d86>] dump_stack+0x45/0x56
[Wed Apr  3 17:47:40 2019]  [<ffffffff811757e3>] print_bad_pte+0x1a3/0x250
[Wed Apr  3 17:47:40 2019]  [<ffffffff8117a73b>] handle_mm_fault+0xebb/0xf00
[Wed Apr  3 17:47:40 2019]  [<ffffffff8172cdd4>] __do_page_fault+0x184/0x560
[Wed Apr  3 17:47:40 2019]  [<ffffffffa02458c9>] ? xfs_file_aio_read+0x179/0x300 [xfs]
[Wed Apr  3 17:47:40 2019]  [<ffffffff811bd41a>] ? do_sync_read+0x5a/0x90
[Wed Apr  3 17:47:40 2019]  [<ffffffff8172d1ca>] do_page_fault+0x1a/0x70
[Wed Apr  3 17:47:40 2019]  [<ffffffff81729628>] page_fault+0x28/0x30

In restic logs:

open repository
repository 33f14e42 opened successfully, password is correct
lock repository
load index files
unexpected fault address 0xc04d22c000
fatal error: fault
[signal SIGBUS: bus error code=0x2 addr=0xc04d22c000 pc=0x458e0e]

goroutine 41 [running]:
runtime.throw(0xc128e9, 0x5)
        /usr/local/go/src/runtime/panic.go:608 +0x72 fp=0xc0071dd5c0 sp=0xc0071dd590 pc=0x42aaa2
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_unix.go:387 +0x2d7 fp=0xc0071dd610 sp=0xc0071dd5c0 pc=0x43fc97
runtime.memmove(0xc04d22c000, 0xc04d14a000, 0x1fe00)
        /usr/local/go/src/runtime/memmove_amd64.s:105 +0xbe fp=0xc0071dd618 sp=0xc0071dd610 pc=0x458e0e
bytes.(*Buffer).grow(0xc0002425b0, 0x200, 0x10000)
        /usr/local/go/src/bytes/buffer.go:145 +0x20a fp=0xc0071dd668 sp=0xc0071dd618 pc=0x4f0f8a
bytes.(*Buffer).ReadFrom(0xc0002425b0, 0x7f1c3231fe60, 0xc04ba361c0, 0x20, 0xc049b3e4e0, 0xc01513a080)
        /usr/local/go/src/bytes/buffer.go:204 +0x4b fp=0xc0071dd6d8 sp=0xc0071dd668 pc=0x4f137b
io/ioutil.readAll(0x7f1c3231fe60, 0xc04ba361c0, 0x200, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/io/ioutil/ioutil.go:36 +0xb5 fp=0xc0071dd720 sp=0xc0071dd6d8 pc=0x56ab45
io/ioutil.ReadAll(0x7f1c3231fe60, 0xc04ba361c0, 0xc0071dd7b0, 0x409a38, 0xb3dfa0, 0xb8bec0, 0xc04cdc7900)
        /usr/local/go/src/io/ioutil/ioutil.go:45 +0x3e fp=0xc0071dd770 sp=0xc0071dd720 pc=0x56ac0e
github.com/restic/restic/internal/backend.LoadAll.func1(0x7f1c3231fe60, 0xc04ba361c0, 0xc04ba361c0, 0x7f1c3231fe60)
        /restic/internal/backend/utils.go:14 +0x3e fp=0xc0071dd7c0 sp=0xc0071dd770 pc=0x76612e
github.com/restic/restic/internal/cache.(*Backend).Load(0xc00020e840, 0xceb960, 0xc0000c28c0, 0xc129bb, 0x5, 0xc04cdc7980, 0x40, 0x0, 0x0, 0xc04cca9f30, ...)
        /restic/internal/cache/backend.go:167 +0xace fp=0xc0071dd9c8 sp=0xc0071dd7c0 pc=0x76b5ee
github.com/restic/restic/internal/backend.LoadAll(0xceb960, 0xc0000c28c0, 0xcf0300, 0xc00020e840, 0xc129bb, 0x5, 0xc04cdc7980, 0x40, 0xfd1347fab246ca8e, 0x2e061a1fe329f1ca, ...)
        /restic/internal/backend/utils.go:13 +0xe2 fp=0xc0071dda40 sp=0xc0071dd9c8 pc=0x765322
github.com/restic/restic/internal/repository.(*Repository).LoadAndDecrypt(0xc000230770, 0xceb960, 0xc0000c28c0, 0xc129bb, 0x5, 0xeca3faaac4b9b49b, 0x12cc5adf059eb21c, 0x1fbf59e24a1fc91e, 0x7b4fd7fd58f2e9d, 0x0, ...)
        /restic/internal/repository/repository.go:75 +0x13f fp=0xc0071ddc10 sp=0xc0071dda40 pc=0x77eabf
github.com/restic/restic/internal/repository.LoadIndexWithDecoder(0xceb960, 0xc0000c28c0, 0xcf44a0, 0xc000230770, 0xeca3faaac4b9b49b, 0x12cc5adf059eb21c, 0x1fbf59e24a1fc91e, 0x7b4fd7fd58f2e9d, 0xc45878, 0x40, ...)
        /restic/internal/repository/index.go:555 +0xcf fp=0xc0071ddce0 sp=0xc0071ddc10 pc=0x778edf
github.com/restic/restic/internal/repository.LoadIndex(0xceb960, 0xc0000c28c0, 0xcf44a0, 0xc000230770, 0xeca3faaac4b9b49b, 0x12cc5adf059eb21c, 0x1fbf59e24a1fc91e, 0x7b4fd7fd58f2e9d, 0x20, 0x0, ...)
        /restic/internal/repository/repository.go:498 +0x8d fp=0xc0071ddd80 sp=0xc0071ddce0 pc=0x78239d
github.com/restic/restic/internal/repository.(*Repository).LoadIndex.func1(0xceb960, 0xc0000c28c0, 0xeca3faaac4b9b49b, 0x12cc5adf059eb21c, 0x1fbf59e24a1fc91e, 0x7b4fd7fd58f2e9d, 0x0, 0x0)
        /restic/internal/repository/repository.go:398 +0x85 fp=0xc0071dde60 sp=0xc0071ddd80 pc=0x7855b5
github.com/restic/restic/internal/repository.ParallelWorkFuncParseID.func1(0xceb960, 0xc0000c28c0, 0xc0023223db, 0x40, 0x0, 0x0)
        /restic/internal/repository/parallel.go:63 +0x11a fp=0xc0071ddf18 sp=0xc0071dde60 pc=0x7854fa
github.com/restic/restic/internal/repository.FilesInParallel.func2(0x8, 0xc46890)
        /restic/internal/repository/parallel.go:40 +0xaa fp=0xc0071ddf88 sp=0xc0071ddf18 pc=0x78539a
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc000206630, 0xc0002066f0)
        /restic/vendor/golang.org/x/sync/errgroup/errgroup.go:58 +0x57 fp=0xc0071ddfd0 sp=0xc0071ddf88 pc=0x769c97
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0071ddfd8 sp=0xc0071ddfd0 pc=0x457c71
created by golang.org/x/sync/errgroup.(*Group).Go
        /restic/vendor/golang.org/x/sync/errgroup/errgroup.go:55 +0x66

goroutine 1 [chan receive]:
github.com/restic/restic/internal/repository.(*Repository).LoadIndex(0xc000230770, 0xceb960, 0xc000040800, 0x0, 0x0)
        /restic/internal/repository/repository.go:419 +0x17d
main.runBackup(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /restic/cmd/restic/cmd_backup.go:445 +0x735
main.glob..func2(0x11d1f80, 0xc00024c500, 0x1, 0x8, 0x0, 0x0)
        /restic/cmd/restic/cmd_backup.go:60 +0x1cf
github.com/spf13/cobra.(*Command).execute(0x11d1f80, 0xc00024c400, 0x8, 0x8, 0x11d1f80, 0xc00024c400)
        /restic/vendor/github.com/spf13/cobra/command.go:762 +0x473
github.com/spf13/cobra.(*Command).ExecuteC(0x11d3280, 0xae3c40, 0x1, 0xc0000782f0)
        /restic/vendor/github.com/spf13/cobra/command.go:852 +0x2fd
github.com/spf13/cobra.(*Command).Execute(0x11d3280, 0xc0001c1ef8, 0xae3c40)
        /restic/vendor/github.com/spf13/cobra/command.go:800 +0x2b
main.main()
        /restic/cmd/restic/main.go:86 +0xd0

goroutine 18 [syscall]:
os/signal.signal_recv(0x0)
        /usr/local/go/src/runtime/sigqueue.go:139 +0x9c
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
        /usr/local/go/src/os/signal/signal_unix.go:29 +0x41

goroutine 19 [chan receive]:
github.com/restic/restic/internal/restic.init.0.func1.1()
        /restic/internal/restic/lock.go:257 +0xb3
created by github.com/restic/restic/internal/restic.init.0.func1
        /restic/internal/restic/lock.go:254 +0x35

goroutine 21 [chan receive]:
github.com/restic/restic/internal/restic.init.2.func1(0xc0000a2180)
        /restic/internal/restic/progress_unix.go:17 +0x63
created by github.com/restic/restic/internal/restic.init.2
        /restic/internal/restic/progress_unix.go:16 +0xab

goroutine 23 [chan receive]:
main.CleanupHandler(0xc0000a2540)
        /restic/cmd/restic/cleanup.go:62 +0x5d
created by main.init.0
        /restic/cmd/restic/cleanup.go:24 +0x67

goroutine 5 [select]:
github.com/restic/restic/internal/ui/termstatus.(*Terminal).run(0xc000084900, 0xceb960, 0xc000040a40)
        /restic/internal/ui/termstatus/status.go:98 +0x16e
github.com/restic/restic/internal/ui/termstatus.(*Terminal).Run(0xc000084900, 0xceb960, 0xc000040a40)
        /restic/internal/ui/termstatus/status.go:83 +0x76
main.glob..func2.1(0x70d469, 0xc0000848a0)
        /restic/cmd/restic/cmd_backup.go:58 +0x6d
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000274050, 0xc00000a500)
        /restic/vendor/gopkg.in/tomb.v2/tomb.go:163 +0x2b
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        /restic/vendor/gopkg.in/tomb.v2/tomb.go:159 +0xba

goroutine 24 [IO wait]:
internal/poll.runtime_pollWait(0x7f1c32314f00, 0x72, 0xc000061488)
        /usr/local/go/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc0000b72d8, 0x72, 0xffffffffffffff01, 0xce4ec0, 0x11884e0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc0000b72d8, 0xc000292001, 0x1000, 0x1000)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc0000b72c0, 0xc000292000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:169 +0x179
os.(*File).read(0xc0000b41c8, 0xc000292000, 0x1000, 0x1000, 0x0, 0x0, 0xc000061590)
        /usr/local/go/src/os/file_unix.go:249 +0x4e
os.(*File).Read(0xc0000b41c8, 0xc000292000, 0x1000, 0x1000, 0x1000, 0x1000, 0x0)
        /usr/local/go/src/os/file.go:108 +0x69
bufio.(*Scanner).Scan(0xc000061730, 0xce7760)
        /usr/local/go/src/bufio/scan.go:213 +0xa6
github.com/restic/restic/internal/backend/sftp.startClient.func1(0xce7760, 0xc0000b41c8, 0xc1172d, 0x3)
        /restic/internal/backend/sftp/sftp.go:53 +0x201
created by github.com/restic/restic/internal/backend/sftp.startClient
        /restic/internal/backend/sftp/sftp.go:51 +0x16a

goroutine 25 [syscall]:
syscall.Syscall6(0xf7, 0x1, 0x7469, 0xc000061db0, 0x1000004, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/syscall/asm_linux_amd64.s:44 +0x5
os.(*Process).blockUntilWaitable(0xc0000acb70, 0x0, 0x0, 0x0)
        /usr/local/go/src/os/wait_waitid.go:31 +0x98
os.(*Process).wait(0xc0000acb70, 0x0, 0x0, 0x0)
        /usr/local/go/src/os/exec_unix.go:22 +0x39
os.(*Process).Wait(0xc0000acb70, 0x0, 0x0, 0x0)
        /usr/local/go/src/os/exec.go:125 +0x2b
os/exec.(*Cmd).Wait(0xc000118c60, 0x0, 0x0)
        /usr/local/go/src/os/exec/exec.go:465 +0x5b
github.com/restic/restic/internal/backend/sftp.startClient.func2(0xc000118c60, 0xc0000b7560)
        /restic/internal/backend/sftp/sftp.go:76 +0x2f
created by github.com/restic/restic/internal/backend/sftp.startClient
        /restic/internal/backend/sftp/sftp.go:75 +0x24f

goroutine 26 [IO wait]:
internal/poll.runtime_pollWait(0x7f1c32314bc0, 0x72, 0xc0000cdc08)
        /usr/local/go/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc0000b7458, 0x72, 0xffffffffffffff01, 0xce4ec0, 0x11884e0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc0000b7458, 0xc00021c101, 0x4, 0x4)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc0000b7440, 0xc00021c168, 0x4, 0x4, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:169 +0x179
os.(*File).read(0xc0000b41e8, 0xc00021c168, 0x4, 0x4, 0x18, 0x0, 0x0)
        /usr/local/go/src/os/file_unix.go:249 +0x4e
os.(*File).Read(0xc0000b41e8, 0xc00021c168, 0x4, 0x4, 0x18, 0x0, 0x0)
        /usr/local/go/src/os/file.go:108 +0x69
io.ReadAtLeast(0xce2e60, 0xc000230700, 0xc00021c168, 0x4, 0x4, 0x4, 0xaef900, 0xc0000cde01, 0xc00021c168)
        /usr/local/go/src/io/io.go:310 +0x88
io.ReadFull(0xce2e60, 0xc000230700, 0xc00021c168, 0x4, 0x4, 0xc0001649c0, 0xc0000cdf70, 0xc0000cdea0)
        /usr/local/go/src/io/io.go:329 +0x58
github.com/pkg/sftp.recvPacket(0xce2e60, 0xc000230700, 0x404c85, 0xc0000b6120, 0xc0000cdf70, 0x1, 0x9fe749, 0xc000243201)
        /restic/vendor/github.com/pkg/sftp/packet.go:143 +0x7b
github.com/pkg/sftp.(*conn).recvPacket(0xc000230700, 0xc0000cdf70, 0xc000000033, 0xc000243288, 0x1, 0x0, 0x0)
        /restic/vendor/github.com/pkg/sftp/conn.go:22 +0x37
github.com/pkg/sftp.(*clientConn).recv(0xc000230700, 0x0, 0x0)
        /restic/vendor/github.com/pkg/sftp/conn.go:64 +0xdc
github.com/pkg/sftp.(*clientConn).loop(0xc000230700)
        /restic/vendor/github.com/pkg/sftp/conn.go:49 +0x5b
created by github.com/pkg/sftp.NewClientPipe
        /restic/vendor/github.com/pkg/sftp/client.go:142 +0x1bd

goroutine 49 [select]:
github.com/restic/restic/internal/ui.(*Backup).Run(0xc00023cc30, 0xceb960, 0xc00021a140, 0x0, 0x0)
        /restic/internal/ui/backup.go:92 +0x2c7
main.runBackup.func2(0x0, 0x0)
        /restic/cmd/restic/cmd_backup.go:423 +0x75
gopkg.in/tomb%2ev2.(*Tomb).run(0xc0002740a0, 0xc000212160)
        /restic/vendor/gopkg.in/tomb.v2/tomb.go:163 +0x2b
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        /restic/vendor/gopkg.in/tomb.v2/tomb.go:159 +0xba

goroutine 35 [select]:
main.refreshLocks(0x11dfee0, 0xc000144480)
        /restic/cmd/restic/lock.go:72 +0x1d0
created by main.lockRepository
        /restic/cmd/restic/lock.go:49 +0x2b3

goroutine 36 [semacquire]:
sync.runtime_Semacquire(0xc000206640)
        /usr/local/go/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc000206638)
        /usr/local/go/src/sync/waitgroup.go:130 +0x64
golang.org/x/sync/errgroup.(*Group).Wait(0xc000206630, 0xc0002066f0, 0xc0000b6900)
        /restic/vendor/golang.org/x/sync/errgroup/errgroup.go:41 +0x31
github.com/restic/restic/internal/repository.FilesInParallel(0xceb960, 0xc0000c28c0, 0x7f1c3231fe40, 0xc00020e840, 0xc129bb, 0x5, 0x4, 0xc00020a260, 0x0, 0x0)
        /restic/internal/repository/parallel.go:49 +0x211
github.com/restic/restic/internal/repository.(*Repository).LoadIndex.func2(0xc0001444e0, 0xc0000b6840, 0xceb960, 0xc000040800, 0xc000230770, 0xc0002143a0)
        /restic/internal/repository/repository.go:414 +0xe9
created by github.com/restic/restic/internal/repository.(*Repository).LoadIndex
        /restic/internal/repository/repository.go:412 +0x10d

goroutine 37 [select]:
github.com/restic/restic/internal/repository.FilesInParallel.func1.1(0x4a2bc, 0xc00232219b, 0x40, 0x40, 0xc035d33318)
        /restic/internal/repository/parallel.go:29 +0xdc
github.com/restic/restic/internal/backend.(*RetryBackend).List.func1.1(0x4a2bc, 0xc00232219b, 0x40, 0x40, 0x0)
        /restic/internal/backend/backend_retry.go:133 +0xd8
github.com/restic/restic/internal/backend/sftp.(*SFTP).List(0xc000288090, 0xceb960, 0xc0000c28c0, 0xc129bb, 0x5, 0xc0000a9500, 0x73c601, 0xc0000a9500)
        /restic/internal/backend/sftp/sftp.go:462 +0x28f
github.com/restic/restic/internal/backend.(*RetryBackend).List.func1(0xc0000c07a0, 0xce5d60)
        /restic/internal/backend/backend_retry.go:127 +0x119
github.com/cenkalti/backoff.RetryNotify(0xc0000d1e50, 0xce5d60, 0xc0000c07a0, 0xc0000d1d98, 0xc0000c07a0, 0xc0000b2ba0)
        /restic/vendor/github.com/cenkalti/backoff/retry.go:37 +0xa2
github.com/restic/restic/internal/backend.(*RetryBackend).retry(0xc00029e140, 0xceb960, 0xc00021a200, 0xc0000b0900, 0xb, 0xc0000d1e50, 0xb, 0x0)
        /restic/internal/backend/backend_retry.go:36 +0x178
github.com/restic/restic/internal/backend.(*RetryBackend).List(0xc00029e140, 0xceb960, 0xc0000c28c0, 0xc129bb, 0x5, 0xc0000c0760, 0x0, 0x0)
        /restic/internal/backend/backend_retry.go:126 +0x235
github.com/restic/restic/internal/repository.FilesInParallel.func1(0x0, 0x0)
        /restic/internal/repository/parallel.go:28 +0x10d
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc000206630, 0xc0000c2940)
        /restic/vendor/golang.org/x/sync/errgroup/errgroup.go:58 +0x57
created by golang.org/x/sync/errgroup.(*Group).Go
        /restic/vendor/golang.org/x/sync/errgroup/errgroup.go:55 +0x66

goroutine 38 [runnable]:
syscall.Syscall(0x0, 0x6, 0xc04d349e00, 0x2000, 0x2000, 0x2000, 0x0)
        /usr/local/go/src/syscall/asm_linux_amd64.s:18 +0x5
syscall.read(0x6, 0xc04d349e00, 0x2000, 0x2000, 0xc00021a001, 0x0, 0x0)
        /usr/local/go/src/syscall/zsyscall_linux_amd64.go:732 +0x5a
syscall.Read(0x6, 0xc04d349e00, 0x2000, 0x2000, 0x0, 0x3e00, 0xc04d348000)
        /usr/local/go/src/syscall/syscall_unix.go:172 +0x49
internal/poll.(*FD).Read(0xc000164c00, 0xc04d349e00, 0x2000, 0x2000, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:165 +0x119
os.(*File).read(0xc00000c0c0, 0xc04d349e00, 0x2000, 0x2000, 0x4f15b1, 0xae3f00, 0x3e00)
        /usr/local/go/src/os/file_unix.go:249 +0x4e
os.(*File).Read(0xc00000c0c0, 0xc04d349e00, 0x2000, 0x2000, 0x3e00, 0x3e00, 0xc04d348000)
        /usr/local/go/src/os/file.go:108 +0x69
bytes.(*Buffer).ReadFrom(0xc000176fc0, 0x7f1c3231fe60, 0xc04d0e98c0, 0x20, 0xc00023d930, 0xc00000c0c0)
        /usr/local/go/src/bytes/buffer.go:206 +0xbd
io/ioutil.readAll(0x7f1c3231fe60, 0xc04d0e98c0, 0x200, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/io/ioutil/ioutil.go:36 +0xb5
io/ioutil.ReadAll(0x7f1c3231fe60, 0xc04d0e98c0, 0xc018c877b0, 0x409a38, 0xb3dfa0, 0xb8bec0, 0xc04d17fb00)
        /usr/local/go/src/io/ioutil/ioutil.go:45 +0x3e
github.com/restic/restic/internal/backend.LoadAll.func1(0x7f1c3231fe60, 0xc04d0e98c0, 0xc04d0e98c0, 0x7f1c3231fe60)
        /restic/internal/backend/utils.go:14 +0x3e
github.com/restic/restic/internal/cache.(*Backend).Load(0xc00020e840, 0xceb960, 0xc0000c28c0, 0xc129bb, 0x5, 0xc04d17fb00, 0x40, 0x0, 0x0, 0xc04d119a40, ...)
        /restic/internal/cache/backend.go:167 +0xace
github.com/restic/restic/internal/backend.LoadAll(0xceb960, 0xc0000c28c0, 0xcf0300, 0xc00020e840, 0xc129bb, 0x5, 0xc04d17fb00, 0x40, 0x29ce3a01275534f, 0x42a43014be9628e3, ...)
        /restic/internal/backend/utils.go:13 +0xe2
github.com/restic/restic/internal/repository.(*Repository).LoadAndDecrypt(0xc000230770, 0xceb960, 0xc0000c28c0, 0xc129bb, 0x5, 0x56702c92067d6f1d, 0x5117629ce6fc1a42, 0x442fd06a02c5231b, 0xe27a51fe70832037, 0x0, ...)
        /restic/internal/repository/repository.go:75 +0x13f
github.com/restic/restic/internal/repository.LoadIndexWithDecoder(0xceb960, 0xc0000c28c0, 0xcf44a0, 0xc000230770, 0x56702c92067d6f1d, 0x5117629ce6fc1a42, 0x442fd06a02c5231b, 0xe27a51fe70832037, 0xc45878, 0x40, ...)
        /restic/internal/repository/index.go:555 +0xcf
github.com/restic/restic/internal/repository.LoadIndex(0xceb960, 0xc0000c28c0, 0xcf44a0, 0xc000230770, 0x56702c92067d6f1d, 0x5117629ce6fc1a42, 0x442fd06a02c5231b, 0xe27a51fe70832037, 0x20, 0x0, ...)
        /restic/internal/repository/repository.go:498 +0x8d
github.com/restic/restic/internal/repository.(*Repository).LoadIndex.func1(0xceb960, 0xc0000c28c0, 0x56702c92067d6f1d, 0x5117629ce6fc1a42, 0x442fd06a02c5231b, 0xe27a51fe70832037, 0x0, 0x0)
        /restic/internal/repository/repository.go:398 +0x85
github.com/restic/restic/internal/repository.ParallelWorkFuncParseID.func1(0xceb960, 0xc0000c28c0, 0xc00232237b, 0x40, 0x0, 0x0)
        /restic/internal/repository/parallel.go:63 +0x11a
github.com/restic/restic/internal/repository.FilesInParallel.func2(0x8, 0xc46890)
        /restic/internal/repository/parallel.go:40 +0xaa
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc000206630, 0xc000206660)
        /restic/vendor/golang.org/x/sync/errgroup/errgroup.go:58 +0x57
created by golang.org/x/sync/errgroup.(*Group).Go
        /restic/vendor/golang.org/x/sync/errgroup/errgroup.go:55 +0x66
goroutine 39 [runnable]:
reflect.Value.Field(0xba1a00, 0xc04d33aa10, 0x199, 0x3, 0x6, 0x55e52, 0x101)
        /usr/local/go/src/reflect/value.go:810 +0x148
encoding/json.(*decodeState).object(0xc04c8a6fa0, 0xba1a00, 0xc04d33aa10, 0x199, 0xc04c8a6fc8, 0x7b)
        /usr/local/go/src/encoding/json/decode.go:745 +0x326
encoding/json.(*decodeState).value(0xc04c8a6fa0, 0xba1a00, 0xc04d33aa10, 0x199, 0xba1a00, 0xc04d33aa10)
        /usr/local/go/src/encoding/json/decode.go:383 +0x6d
encoding/json.(*decodeState).array(0xc04c8a6fa0, 0xad44e0, 0xc04d30d160, 0x197, 0xc04c8a6fc8, 0x5b)
        /usr/local/go/src/encoding/json/decode.go:573 +0x1b3
encoding/json.(*decodeState).value(0xc04c8a6fa0, 0xad44e0, 0xc04d30d160, 0x197, 0xb66520, 0xc04d30d160)
        /usr/local/go/src/encoding/json/decode.go:373 +0xf9
encoding/json.(*decodeState).object(0xc04c8a6fa0, 0xac1460, 0xc04c8afa28, 0x196, 0xc04c8a6fc8, 0x7b)
        /usr/local/go/src/encoding/json/decode.go:781 +0x16fe
encoding/json.(*decodeState).value(0xc04c8a6fa0, 0xac1460, 0xc04c8afa28, 0x196, 0xac1460, 0xc04c8afa28)
        /usr/local/go/src/encoding/json/decode.go:383 +0x6d
encoding/json.(*decodeState).array(0xc04c8a6fa0, 0xad02a0, 0xc015eeb398, 0x197, 0xc04c8a6fc8, 0x5b)
        /usr/local/go/src/encoding/json/decode.go:573 +0x1b3
encoding/json.(*decodeState).value(0xc04c8a6fa0, 0xad02a0, 0xc015eeb398, 0x197, 0xb66480, 0xc015eeb398)
        /usr/local/go/src/encoding/json/decode.go:373 +0xf9
encoding/json.(*decodeState).object(0xc04c8a6fa0, 0xac1420, 0xc015eeb380, 0x16, 0xc04c8a6fc8, 0x7b)
        /usr/local/go/src/encoding/json/decode.go:781 +0x16fe
encoding/json.(*decodeState).value(0xc04c8a6fa0, 0xac1420, 0xc015eeb380, 0x16, 0xc0071e18f8, 0x6fe177)
        /usr/local/go/src/encoding/json/decode.go:383 +0x6d
encoding/json.(*decodeState).unmarshal(0xc04c8a6fa0, 0xac1420, 0xc015eeb380, 0xc04c8a6fc8, 0x0)
        /usr/local/go/src/encoding/json/decode.go:180 +0x204
encoding/json.Unmarshal(0xc04cf02010, 0x4b14e, 0x7fdf0, 0xac1420, 0xc015eeb380, 0xc04cf4d15e, 0x10)
        /usr/local/go/src/encoding/json/decode.go:107 +0x122
github.com/restic/restic/internal/repository.DecodeIndex(0xc04cf02010, 0x4b14e, 0x7fdf0, 0xc129bb, 0x5, 0xcdb3bdf5573d6c92)
        /restic/internal/repository/index.go:461 +0x8b
github.com/restic/restic/internal/repository.LoadIndexWithDecoder(0xceb960, 0xc0000c28c0, 0xcf44a0, 0xc000230770, 0xcdb3bdf5573d6c92, 0x67c9bb71c8835b2c, 0x91e5da1ad8304660, 0xdb68faea048fe3bc, 0xc45878, 0x40, ...)
        /restic/internal/repository/index.go:560 +0x10c
github.com/restic/restic/internal/repository.LoadIndex(0xceb960, 0xc0000c28c0, 0xcf44a0, 0xc000230770, 0xcdb3bdf5573d6c92, 0x67c9bb71c8835b2c, 0x91e5da1ad8304660, 0xdb68faea048fe3bc, 0x20, 0x0, ...)
        /restic/internal/repository/repository.go:498 +0x8d
github.com/restic/restic/internal/repository.(*Repository).LoadIndex.func1(0xceb960, 0xc0000c28c0, 0xcdb3bdf5573d6c92, 0x67c9bb71c8835b2c, 0x91e5da1ad8304660, 0xdb68faea048fe3bc, 0x0, 0x0)
        /restic/internal/repository/repository.go:398 +0x85
github.com/restic/restic/internal/repository.ParallelWorkFuncParseID.func1(0xceb960, 0xc0000c28c0, 0xc0023224fb, 0x40, 0x0, 0x0)
        /restic/internal/repository/parallel.go:63 +0x11a
github.com/restic/restic/internal/repository.FilesInParallel.func2(0x8, 0xc46890)
        /restic/internal/repository/parallel.go:40 +0xaa
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc000206630, 0xc000206690)
        /restic/vendor/golang.org/x/sync/errgroup/errgroup.go:58 +0x57
created by golang.org/x/sync/errgroup.(*Group).Go
        /restic/vendor/golang.org/x/sync/errgroup/errgroup.go:55 +0x66

goroutine 40 [runnable]:
reflect.(*rtype).Name(0xb84720, 0xc04d335340, 0x191)
        /usr/local/go/src/reflect/type.go:874 +0xa7
encoding/json.indirect(0xb84720, 0xc04d335340, 0x191, 0xa9800, 0xa981a, 0x0, 0xc04d06092e, 0x0, 0xae3e00, 0xc04d335300, ...)
        /usr/local/go/src/encoding/json/decode.go:459 +0x5f4
encoding/json.(*decodeState).literalStore(0xc04cdca6e0, 0xc04d06097f, 0x42, 0x67481, 0xb84720, 0xc04d335340, 0x191, 0x18900, 0x475f0, 0xc04cdca708)
        /usr/local/go/src/encoding/json/decode.go:869 +0x1f1
encoding/json.(*decodeState).value(0xc04cdca6e0, 0xb84720, 0xc04d335340, 0x191, 0xba1a00, 0xc04d335340)
        /usr/local/go/src/encoding/json/decode.go:397 +0x20e
encoding/json.(*decodeState).object(0xc04cdca6e0, 0xba1a00, 0xc04d335340, 0x199, 0xc18120, 0xb)
        /usr/local/go/src/encoding/json/decode.go:781 +0x16fe
encoding/json.(*decodeState).value(0xc04cdca6e0, 0xba1a00, 0xc04d335340, 0x199, 0xba1a00, 0xc04d335340)
        /usr/local/go/src/encoding/json/decode.go:383 +0x6d
encoding/json.(*decodeState).array(0xc04cdca6e0, 0xad44e0, 0xc04d2e9860, 0x197, 0xc04cdca708, 0x5b)
        /usr/local/go/src/encoding/json/decode.go:573 +0x1b3
encoding/json.(*decodeState).value(0xc04cdca6e0, 0xad44e0, 0xc04d2e9860, 0x197, 0xb66520, 0xc04d2e9860)
        /usr/local/go/src/encoding/json/decode.go:373 +0xf9
encoding/json.(*decodeState).object(0xc04cdca6e0, 0xac1460, 0xc04ab081e0, 0x196, 0xc04cdca708, 0x7b)
        /usr/local/go/src/encoding/json/decode.go:781 +0x16fe
encoding/json.(*decodeState).value(0xc04cdca6e0, 0xac1460, 0xc04ab081e0, 0x196, 0xac1460, 0xc04ab081e0)
        /usr/local/go/src/encoding/json/decode.go:383 +0x6d
encoding/json.(*decodeState).array(0xc04cdca6e0, 0xad02a0, 0xc007a17758, 0x197, 0xc04cdca708, 0x5b)
        /usr/local/go/src/encoding/json/decode.go:573 +0x1b3
encoding/json.(*decodeState).value(0xc04cdca6e0, 0xad02a0, 0xc007a17758, 0x197, 0xb66480, 0xc007a17758)
        /usr/local/go/src/encoding/json/decode.go:373 +0xf9
encoding/json.(*decodeState).object(0xc04cdca6e0, 0xac1420, 0xc007a17740, 0x16, 0xc04cdca708, 0x7b)
        /usr/local/go/src/encoding/json/decode.go:781 +0x16fe
encoding/json.(*decodeState).value(0xc04cdca6e0, 0xac1420, 0xc007a17740, 0x16, 0xc018c838f8, 0x6fe177)
        /usr/local/go/src/encoding/json/decode.go:383 +0x6d
encoding/json.(*decodeState).unmarshal(0xc04cdca6e0, 0xac1420, 0xc007a17740, 0xc04cdca708, 0x0)
        /usr/local/go/src/encoding/json/decode.go:180 +0x204
encoding/json.Unmarshal(0xc04d048010, 0x475f0, 0x7fdf0, 0xac1420, 0xc007a17740, 0xc04d08f600, 0x10)
        /usr/local/go/src/encoding/json/decode.go:107 +0x122
github.com/restic/restic/internal/repository.DecodeIndex(0xc04d048010, 0x475f0, 0x7fdf0, 0xc129bb, 0x5, 0x8e2d5744ca8521c4)
        /restic/internal/repository/index.go:461 +0x8b
github.com/restic/restic/internal/repository.LoadIndexWithDecoder(0xceb960, 0xc0000c28c0, 0xcf44a0, 0xc000230770, 0x8e2d5744ca8521c4, 0x2c00a43ea734bcf9, 0xf1e4c9b8e93c2d16, 0xc617f5be23d414f3, 0xc45878, 0x40, ...)
        /restic/internal/repository/index.go:560 +0x10c
github.com/restic/restic/internal/repository.LoadIndex(0xceb960, 0xc0000c28c0, 0xcf44a0, 0xc000230770, 0x8e2d5744ca8521c4, 0x2c00a43ea734bcf9, 0xf1e4c9b8e93c2d16, 0xc617f5be23d414f3, 0x20, 0x0, ...)
        /restic/internal/repository/repository.go:498 +0x8d
github.com/restic/restic/internal/repository.(*Repository).LoadIndex.func1(0xceb960, 0xc0000c28c0, 0x8e2d5744ca8521c4, 0x2c00a43ea734bcf9, 0xf1e4c9b8e93c2d16, 0xc617f5be23d414f3, 0x0, 0x0)
        /restic/internal/repository/repository.go:398 +0x85
github.com/restic/restic/internal/repository.ParallelWorkFuncParseID.func1(0xceb960, 0xc0000c28c0, 0xc00232249b, 0x40, 0x0, 0x0)
        /restic/internal/repository/parallel.go:63 +0x11a
github.com/restic/restic/internal/repository.FilesInParallel.func2(0x8, 0xc46890)
        /restic/internal/repository/parallel.go:40 +0xaa
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc000206630, 0xc0002066c0)
        /restic/vendor/golang.org/x/sync/errgroup/errgroup.go:58 +0x57
created by golang.org/x/sync/errgroup.(*Group).Go
        /restic/vendor/golang.org/x/sync/errgroup/errgroup.go:55 +0x66
0 Likes

#2

Yes you should. This error message comes from the kernel and usually is a strong indicator of a hardware problem. (Restic has no direct access to its own page map; this is a kernel structure. This is not a bug in restic.)

Run memtest on this system immediately, and for several hours. I’d wager that it will report problems, which would indicate that the RAM, motherboard, and/or CPU are faulty. (It’s also possible that another component could cause problems, e.g. undervoltage from a faulty PSU.)

Basically you need to swap out components starting with the RAM until the memtest comes back clean after running for several hours.

It is possible that the operating system running on this machine has already been corrupted. When you fix the hardware problem, you should ideally perform a reinstall of the OS from scratch and restore all data from a backup.

1 Like

#3

I’d be really interested if memtest86 turns up anything :slight_smile:

0 Likes

#4

Unfortunately this is not mine. It’s a dedicated from one online provider. I don’t have access to POST.

However this one is a Dell. The omreport tools say nothing wrong:

# omreport chassis memory
Memory Information

Health : Ok

Attributes of Memory Array(s)

Attributes of Memory Array(s)
Location           : System Board or Motherboard
Use                : System Memory
Installed Capacity : 16384  MB
Maximum Capacity   : 16384  MB
Slots Available    : 4
Slots Used         : 4
Error Correction   : Multibit ECC

Total of Memory Array(s)
Total Installed Capacity                     : 16384  MB
Total Installed Capacity Available to the OS : 16037  MB
Total Maximum Capacity                       : 16384  MB

Details of Memory Array 1
Index          : 0
Status         : Ok
Connector Name : DIMM_A1
Type           : DDR3 - Synchronous Unregistered (Unbuffered)
Size           : 4096  MB

Index          : 1
Status         : Ok
Connector Name : DIMM_A2
Type           : DDR3 - Synchronous Unregistered (Unbuffered)
Size           : 4096  MB

Index          : 2
Status         : Ok
Connector Name : DIMM_A3
Type           : DDR3 - Synchronous Unregistered (Unbuffered)
Size           : 4096  MB

Index          : 3
Status         : Ok
Connector Name : DIMM_A4
Type           : DDR3 - Synchronous Unregistered (Unbuffered)
Size           : 4096  MB

And nothing about neither ECC correction nor MCE in dmesg.

0 Likes

#5

It would probably be a good idea to let them know about this. If your machine is a VM, perhaps they can move it to a different physical server.

What kernel version is the server running?

0 Likes

#6

This is physical one.
Kernel is a bit old 3.13.0-44-generic.

Anyway I do not really care about it because I am planning to resign that server once all data would be migrated :slightly_smiling_face:.

I think it’s more kernel related than HW.

0 Likes

#7

Once you get the data moved off, I’d be sure to run restic check --read-data on all repositories to make sure that any possible memory issues on the server didn’t corrupt repository data!

0 Likes

#8

Did it.

Took 10h:

using temporary cache in /srv/restic-check/restic-check-cache-650786045
repository 33f14e42 opened successfully, password is correct
created new cache in /srv/restic-check/restic-check-cache-650786045
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
read all data
[9:50:17] 100.00%  351750 / 351750 items
duration: 9:50:17
no errors were found 

Guess I am lucky

2 Likes