Stacktrace on restic prune

When I run prune after about an hour of “pruning” I get a stack trace. I am using rclone as a backend, so I suspect this is the issue, but I thought I should report it anyway:

/usr/bin/restic --repo rest:<repo> prune
repository 1ab49f08 opened successfully, password is correct
loading indexes...
loading all snapshots...
finding data that is still in use for 17 snapshots
[1:29] 100.00%  17 / 17 snapshots
searching used packs...
collecting packs for deletion and repacking
[1:22:45] 0.00%  0 / 6694496 packs processed
Get "<repo>": EOF
List
github.com/restic/restic/internal/backend/rest.(*Backend).List
        github.com/restic/restic/internal/backend/rest/rest.go:411
github.com/restic/restic/internal/backend.(*RetryBackend).List.func1
        github.com/restic/restic/internal/backend/backend_retry.go:137
github.com/cenkalti/backoff/v4.RetryNotifyWithTimer
        github.com/cenkalti/backoff/v4@v4.1.1/retry.go:55
github.com/cenkalti/backoff/v4.RetryNotify
        github.com/cenkalti/backoff/v4@v4.1.1/retry.go:34
github.com/restic/restic/internal/backend.(*RetryBackend).retry
        github.com/restic/restic/internal/backend/backend_retry.go:46
github.com/restic/restic/internal/backend.(*RetryBackend).List
        github.com/restic/restic/internal/backend/backend_retry.go:136
github.com/restic/restic/internal/repository.(*Repository).List
        github.com/restic/restic/internal/repository/repository.go:670
main.prune
        github.com/restic/restic/cmd/restic/cmd_prune.go:312
main.runPruneWithRepo
        github.com/restic/restic/cmd/restic/cmd_prune.go:173
main.runPrune
        github.com/restic/restic/cmd/restic/cmd_prune.go:151
main.glob..func19
        github.com/restic/restic/cmd/restic/cmd_prune.go:35
github.com/spf13/cobra.(*Command).execute
        github.com/spf13/cobra@v1.2.1/command.go:856
github.com/spf13/cobra.(*Command).ExecuteC
        github.com/spf13/cobra@v1.2.1/command.go:974
github.com/spf13/cobra.(*Command).Execute
        github.com/spf13/cobra@v1.2.1/command.go:902
main.main
        github.com/restic/restic/cmd/restic/main.go:98
runtime.main
        runtime/proc.go:225
runtime.goexit
        runtime/asm_amd64.s:1371

Thanks.

Which restic version are you using? On which backend is the repository stored? How large is your repository 30-40TB?

restic 0.12.1 compiled with go1.16.6 on linux/amd64
rclone version
rclone v1.56.2

  • os/version: arch (64 bit)
  • os/kernel: 5.14.8-arch1-1 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.17.1
  • go/linking: dynamic
  • go/tags: none

rclone is going to google drive (I know this is not really supported).

My store is 31TB

Hmm, the problem is that the list operation takes ages (restic doesn’t retry it as it took more that 10 minutes) and then fails. Is there maybe an option in rclone the cache the directory listing such that a second prune run could succeed?

There doesn’t seem to be any way to do that. I’ve checked the man page on rclone as well as the setup for google drive.

There seems to be a --fast-list option. Maybe that helps?

I saw that option too, and tried it, but it made no difference…