Load snapshot: The specified key does not exist and FindLatestSnapshot error

I’ve been getting an error about a snapshot not existing and my backups have started to error out. I tried restic forget with that snapshot and verified it’s not being returned in the restics snaphots list and via s3cmd to Digitalocean spaces where it is stored. Is there some where I need to manually remove a reference to this snapshot? All the restic commands continue to check for it.

root@PVG:~# restic version
restic 0.9.6 compiled with go1.13.4 on linux/amd64

root@PVG:~# restic forget 7c81e8fec1
repository e573cdd9 opened successfully, password is correct
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 582.280027ms: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 703.28564ms: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 1.040217184s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 2.002763936s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 1.431768704s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 2.492721545s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 3.399927963s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 6.842164996s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 13.00939464s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 16.927710407s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 391.517075ms: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 594.826393ms: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 1.326470261s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 1.212558276s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 1.779941781s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 3.268621161s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 6.097818893s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 11.639721773s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 16.05103845s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 23.002952479s: The specified key does not exist.
Ignoring "7c81e8fec1c24c816f98a32bfb1b209c3407b61743e6934af17b6ae4a30e60b1", could not load snapshot: The specified key does not exist.

root@PVG:~#  s3cmd get s3://<removed>/pvg/snapshots/7c81e8fec1c24c816f98a32bfb1b209c3407b61743e6934af17b6ae4a30e60b1
download: 's3://removed/pvg/snapshots/7c81e8fec1c24c816f98a32bfb1b209c3407b61743e6934af17b6ae4a30e60b1' -> './7c81e8fec1c24c816f98a32bfb1b209c3407b61743e6934af17b6ae4a30e60b1'  [1 of 1]
ERROR: S3 error: 404 (NoSuchKey)

root@PVG:~# restic check --cleanup-cache
using temporary cache in /tmp/restic-check-cache-821615830
repository e573cdd9 opened successfully, password is correct
created new cache in /tmp/restic-check-cache-821615830
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 507.606314ms: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 985.229971ms: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 803.546856ms: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 1.486109007s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 2.070709754s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 3.67875363s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 4.459624189s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 6.775444383s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 15.10932531s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 16.547394626s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 396.557122ms: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 597.811922ms: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 1.409144665s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 1.192358242s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 3.456004252s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 4.543793083s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 5.830976587s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 4.513276731s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 8.436116856s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 21.283270947s: The specified key does not exist.
error: The specified key does not exist.
Fatal: repository contains errors

root@PVG:~# restic backup -q /srv/users
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 582.280027ms: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 703.28564ms: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 1.040217184s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 2.002763936s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 1.431768704s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 2.492721545s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 3.399927963s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 6.842164996s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 13.00939464s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 16.927710407s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 391.517075ms: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 594.826393ms: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 1.326470261s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 1.212558276s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 1.779941781s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 3.268621161s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 6.097818893s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 11.639721773s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 16.05103845s: The specified key does not exist.
Load(<snapshot/7c81e8fec1>, 0, 0) returned error, retrying after 23.002952479s: The specified key does not exist.
Error loading snapshot 7c81e8fe: The specified key does not exist.
github.com/restic/restic/internal/restic.FindLatestSnapshot.func1
        /restic/internal/restic/snapshot_find.go:39
github.com/restic/restic/internal/repository.(*Repository).List.func1
        /restic/internal/repository/repository.go:644
github.com/restic/restic/internal/backend.(*RetryBackend).List.func1.1
        /restic/internal/backend/backend_retry.go:133
github.com/restic/restic/internal/backend/s3.(*Backend).List
        /restic/internal/backend/s3/s3.go:433
github.com/restic/restic/internal/backend.(*RetryBackend).List.func1
        /restic/internal/backend/backend_retry.go:127
github.com/cenkalti/backoff.RetryNotify
        /restic/vendor/github.com/cenkalti/backoff/retry.go:37
github.com/restic/restic/internal/backend.(*RetryBackend).retry
        /restic/internal/backend/backend_retry.go:36
github.com/restic/restic/internal/backend.(*RetryBackend).List
        /restic/internal/backend/backend_retry.go:126
github.com/restic/restic/internal/repository.(*Repository).List
        /restic/internal/repository/repository.go:638
github.com/restic/restic/internal/restic.FindLatestSnapshot
        /restic/internal/restic/snapshot_find.go:36
main.findParentSnapshot
        /restic/cmd/restic/cmd_backup.go:377
main.runBackup
        /restic/cmd/restic/cmd_backup.go:495
main.glob..func2
        /restic/cmd/restic/cmd_backup.go:63
github.com/spf13/cobra.(*Command).execute
        /restic/vendor/github.com/spf13/cobra/command.go:762
github.com/spf13/cobra.(*Command).ExecuteC
        /restic/vendor/github.com/spf13/cobra/command.go:852
github.com/spf13/cobra.(*Command).Execute
        /restic/vendor/github.com/spf13/cobra/command.go:800
main.main
        /restic/cmd/restic/main.go:86
runtime.main
        /usr/local/go/src/runtime/proc.go:203
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1357

I reached out to Digitalocean, Turns out this was a problem with spaces. Apparently whatever index they are referring to is being checked by restic at some point, but when running the restic snapshot command it wasn’t being listed like it wasn’t with s3cmd.

Our engineers found that the file was removed/deleted a while ago and the index for the bucket still had the file. This is why the file would list but otherwise act like it’s not there. Our engineering team has repaired the index for the bucket and your object //pvg/snapshots/7c81e8fec1c24c816f98a32bfb1b209c3407b61743e6934af17b6ae4a30e60b1 shouldn’t be listing anymore.