Possibly bug - load lock reports not exist but it is there in backend

1

when I do restic unlock command - I am getting following output:

repository 1c8e9890 opened successfully, password is correct
Load(<lock/c1e2836ea0>, 0, 0) returned error, retrying after 720.254544ms: <lock/c1e2836ea0> does not exist
Load(<lock/4cec06c655>, 0, 0) returned error, retrying after 582.280027ms: <lock/4cec06c655> does not exist
Load(<lock/4cec06c655>, 0, 0) returned error, retrying after 703.28564ms: <lock/4cec06c655> does not exist
Load(<lock/c1e2836ea0>, 0, 0) returned error, retrying after 693.478123ms: <lock/c1e2836ea0> does not exist
Load(<lock/4cec06c655>, 0, 0) returned error, retrying after 1.335175957s: <lock/4cec06c655> does not exist
Load(<lock/c1e2836ea0>, 0, 0) returned error, retrying after 636.341646ms: <lock/c1e2836ea0> does not exist
Load(<lock/c1e2836ea0>, 0, 0) returned error, retrying after 1.107876242s: <lock/c1e2836ea0> does not exist
Load(<lock/4cec06c655>, 0, 0) returned error, retrying after 1.007386063s: <lock/4cec06c655> does not exist
Load(<lock/c1e2836ea0>, 0, 0) returned error, retrying after 2.027308147s: <lock/c1e2836ea0> does not exist
Load(<lock/4cec06c655>, 0, 0) returned error, retrying after 2.569756966s: <lock/4cec06c655> does not exist
Load(<lock/c1e2836ea0>, 0, 0) returned error, retrying after 4.987726727s: <lock/c1e2836ea0> does not exist
Load(<lock/4cec06c655>, 0, 0) returned error, retrying after 2.711970641s: <lock/4cec06c655> does not exist
Load(<lock/4cec06c655>, 0, 0) returned error, retrying after 5.015617898s: <lock/4cec06c655> does not exist
Load(<lock/c1e2836ea0>, 0, 0) returned error, retrying after 4.659096946s: <lock/c1e2836ea0> does not exist
Load(<lock/4cec06c655>, 0, 0) returned error, retrying after 8.277195667s: <lock/4cec06c655> does not exist
Load(<lock/c1e2836ea0>, 0, 0) returned error, retrying after 6.689436284s: <lock/c1e2836ea0> does not exist
Load(<lock/c1e2836ea0>, 0, 0) returned error, retrying after 10.163166574s: <lock/c1e2836ea0> does not exist
Load(<lock/4cec06c655>, 0, 0) returned error, retrying after 15.10932531s: <lock/4cec06c655> does not exist
Load(<lock/c1e2836ea0>, 0, 0) returned error, retrying after 13.811796615s: <lock/c1e2836ea0> does not exist
Load(<lock/4cec06c655>, 0, 0) returned error, retrying after 13.516432903s: <lock/4cec06c655> does not exist
successfully removed locks

key points - says load lock does not exist but ends with successfully removed locks.

but if I restic list locks immediately after; I get this

repository 1c8e9890 opened successfully, password is correct
Load(<lock/4cec06c655>, 0, 0) returned error, retrying after 720.254544ms: <lock/4cec06c655> does not exist
Load(<lock/c1e2836ea0>, 0, 0) returned error, retrying after 582.280027ms: <lock/c1e2836ea0> does not exist
Load(<lock/c1e2836ea0>, 0, 0) returned error, retrying after 703.28564ms: <lock/c1e2836ea0> does not exist
Load(<lock/4cec06c655>, 0, 0) returned error, retrying after 693.478123ms: <lock/4cec06c655> does not exist
Load(<lock/c1e2836ea0>, 0, 0) returned error, retrying after 1.335175957s: <lock/c1e2836ea0> does not exist
Load(<lock/4cec06c655>, 0, 0) returned error, retrying after 636.341646ms: <lock/4cec06c655> does not exist
Load(<lock/4cec06c655>, 0, 0) returned error, retrying after 1.107876242s: <lock/4cec06c655> does not exist
Load(<lock/c1e2836ea0>, 0, 0) returned error, retrying after 1.007386063s: <lock/c1e2836ea0> does not exist
Load(<lock/4cec06c655>, 0, 0) returned error, retrying after 2.027308147s: <lock/4cec06c655> does not exist
Load(<lock/c1e2836ea0>, 0, 0) returned error, retrying after 2.569756966s: <lock/c1e2836ea0> does not exist
Load(<lock/4cec06c655>, 0, 0) returned error, retrying after 4.987726727s: <lock/4cec06c655> does not exist
Load(<lock/c1e2836ea0>, 0, 0) returned error, retrying after 2.711970641s: <lock/c1e2836ea0> does not exist
Load(<lock/c1e2836ea0>, 0, 0) returned error, retrying after 5.015617898s: <lock/c1e2836ea0> does not exist
Load(<lock/4cec06c655>, 0, 0) returned error, retrying after 4.659096946s: <lock/4cec06c655> does not exist
Load(<lock/c1e2836ea0>, 0, 0) returned error, retrying after 8.277195667s: <lock/c1e2836ea0> does not exist
Load(<lock/4cec06c655>, 0, 0) returned error, retrying after 6.689436284s: <lock/4cec06c655> does not exist
Load(<lock/4cec06c655>, 0, 0) returned error, retrying after 10.163166574s: <lock/4cec06c655> does not exist
Load(<lock/c1e2836ea0>, 0, 0) returned error, retrying after 15.10932531s: <lock/c1e2836ea0> does not exist
Load(<lock/4cec06c655>, 0, 0) returned error, retrying after 13.811796615s: <lock/4cec06c655> does not exist
Load(<lock/c1e2836ea0>, 0, 0) returned error, retrying after 13.516432903s: <lock/c1e2836ea0> does not exist
Load(<lock/c1e2836ea0>, 0, 0) returned error, retrying after 430.435708ms: <lock/c1e2836ea0> does not exist
Load(<lock/4cec06c655>, 0, 0) returned error, retrying after 535.336638ms: <lock/4cec06c655> does not exist
Load(<lock/c1e2836ea0>, 0, 0) returned error, retrying after 1.021868578s: <lock/c1e2836ea0> does not exist
Load(<lock/4cec06c655>, 0, 0) returned error, retrying after 594.835683ms: <lock/4cec06c655> does not exist
Load(<lock/4cec06c655>, 0, 0) returned error, retrying after 896.717884ms: <lock/4cec06c655> does not exist
Load(<lock/c1e2836ea0>, 0, 0) returned error, retrying after 1.409144665s: <lock/c1e2836ea0> does not exist
Load(<lock/4cec06c655>, 0, 0) returned error, retrying after 1.192358242s: <lock/4cec06c655> does not exist
Load(<lock/c1e2836ea0>, 0, 0) returned error, retrying after 2.304002835s: <lock/c1e2836ea0> does not exist
Load(<lock/4cec06c655>, 0, 0) returned error, retrying after 3.029195388s: <lock/4cec06c655> does not exist
Load(<lock/c1e2836ea0>, 0, 0) returned error, retrying after 2.59154515s: <lock/c1e2836ea0> does not exist
Load(<lock/4cec06c655>, 0, 0) returned error, retrying after 2.005900769s: <lock/4cec06c655> does not exist
Load(<lock/c1e2836ea0>, 0, 0) returned error, retrying after 2.499590179s: <lock/c1e2836ea0> does not exist
Load(<lock/4cec06c655>, 0, 0) returned error, retrying after 6.306154355s: <lock/4cec06c655> does not exist
Load(<lock/c1e2836ea0>, 0, 0) returned error, retrying after 8.401962033s: <lock/c1e2836ea0> does not exist
Load(<lock/4cec06c655>, 0, 0) returned error, retrying after 4.950254196s: <lock/4cec06c655> does not exist
Load(<lock/c1e2836ea0>, 0, 0) returned error, retrying after 9.352915637s: <lock/c1e2836ea0> does not exist
Load(<lock/4cec06c655>, 0, 0) returned error, retrying after 7.164825377s: <lock/4cec06c655> does not exist
Load(<lock/4cec06c655>, 0, 0) returned error, retrying after 22.912714797s: <lock/4cec06c655> does not exist
Load(<lock/c1e2836ea0>, 0, 0) returned error, retrying after 10.271074824s: <lock/c1e2836ea0> does not exist
Load(<lock/c1e2836ea0>, 0, 0) returned error, retrying after 12.941307974s: <lock/c1e2836ea0> does not exist
4cec06c6559a0b43d6e6fba709a2871d8977db262bd05bf0a30956fa93c6ded6
c1e2836ea022fac11fa6857708b21046b6154580dc05e40735722bfa114d3dcf
c386c8eec6844c06747573f7990803cd83c9c3adf16b2414068f992f0cbd0992

key points:

at the end - the locks are clearly there - but says can’t be loaded? also these are the same locks deleted before.

the backups/prunes seems to be proceeding ok but all of these weird lock messages make me worried so not sure what to do and any insights appreciated.

2

What backend are you using?

3

A full command line (with private details redacted) and output of restic version is necessary to help with questions.

4

On the backend side - it is an S3 compatible backend (idrive e2) accessed through rclone.

restic version gives the following:

restic 0.12.0 compiled with go1.15.8 on linux/amd64

the command line for the unlocking / lock listing is the simple commands below wiithout any more command options

restic unlock -r [target]
restic list locks [target]
5

There have been a fair few fixes, including improved error reporting for rclone backends and around locking specifically, since 0.12. list locks also would incorrectly create locks with that version. In addition there have been plenty of performance enhancements and support for compression. It may be worth updating and re-trying. I’d also suggest checking your backend storage logs to see if there are any relevant entries.

6

That looks like a bug of the S3 storage. Please try to manually access these files, I’d expect that this fails too.

7

Updating to 0.14 didn’t seem to fix things.

Given error doesn’t seem to cause problems (as per messages; there is errors but the processes continue) - I just manually deleted the lock files in the S3 backend and things work as normal now

Guess won’t be able to hunt down why it was happening but at least have benefit of new restic version to play with…