Key error while acess, maybe lock issue

Getting Help
1

Hi,

since a week I see this lock related output while running restic commands:

$ restic snapshots
password is correct
Load(<lock/93c4e71b21>, 0, 0) returned error, retrying after 268.120154ms: The specified key does not exist.
Load(<lock/93c4e71b21>, 0, 0) returned error, retrying after 377.571483ms: The specified key does not exist.
Load(<lock/93c4e71b21>, 0, 0) returned error, retrying after 1.341156292s: The specified key does not exist.
Load(<lock/93c4e71b21>, 0, 0) returned error, retrying after 1.133408837s: The specified key does not exist.
Load(<lock/93c4e71b21>, 0, 0) returned error, retrying after 3.7434438s: The specified key does not exist.
Load(<lock/93c4e71b21>, 0, 0) returned error, retrying after 5.098939595s: The specified key does not exist.
Load(<lock/93c4e71b21>, 0, 0) returned error, retrying after 7.831059838s: The specified key does not exist.
Load(<lock/93c4e71b21>, 0, 0) returned error, retrying after 9.95312954s: The specified key does not exist.
Load(<lock/93c4e71b21>, 0, 0) returned error, retrying after 8.426647113s: The specified key does not exist.
Load(<lock/93c4e71b21>, 0, 0) returned error, retrying after 16.190068529s: The specified key does not exist.
Load(<lock/93c4e71b21>, 0, 0) returned error, retrying after 523.056555ms: The specified key does not exist.
Load(<lock/93c4e71b21>, 0, 0) returned error, retrying after 529.234649ms: The specified key does not exist.
Load(<lock/93c4e71b21>, 0, 0) returned error, retrying after 722.347121ms: The specified key does not exist.
Load(<lock/93c4e71b21>, 0, 0) returned error, retrying after 980.502782ms: The specified key does not exist.
Load(<lock/93c4e71b21>, 0, 0) returned error, retrying after 1.646561404s: The specified key does not exist.
Load(<lock/93c4e71b21>, 0, 0) returned error, retrying after 4.870009247s: The specified key does not exist.
Load(<lock/93c4e71b21>, 0, 0) returned error, retrying after 3.643921449s: The specified key does not exist.
Load(<lock/93c4e71b21>, 0, 0) returned error, retrying after 5.053931703s: The specified key does not exist.
Load(<lock/93c4e71b21>, 0, 0) returned error, retrying after 17.352963537s: The specified key does not exist.
Load(<lock/93c4e71b21>, 0, 0) returned error, retrying after 14.622071531s: The specified key does not exist.
ID        Date                 Host            Tags          Directory
----------------------------------------------------------------------
c024c858  2017-12-22 04:02:20  staging-katuma  postgres  ┌── db_pgsql_openfoodnetwork.sql
                                               database  └── 
5bec210e  2017-12-22 04:12:34  staging-katuma  postgres  ┌── db_pgsql_openfoodnetwork.sql
                                               database  └── 
31482ca6  2017-12-22 04:22:24  staging-katuma  postgres  ┌── db_pgsql_openfoodnetwork.sql
                                               database  └── 
18d3a028  2017-12-22 04:42:30  staging-katuma  postgres  ┌── db_pgsql_openfoodnetwork.sql
                                               database  └── 
89e7328b  2017-12-22 04:48:50  staging-katuma  postgres  ┌── db_pgsql_openfoodnetwork.sql
                                               database  └── 
----------------------------------------------------------------------
5 snapshots

Current version:

restic 0.8.0
compiled with go1.9 on linux/amd64

Snapshots are stored in Digital Ocean Spaces. I didn’t change anything and I never saw this output before, the cron job creating the backups seems to work fine.

Any hint?

2

This is likely caused by the Spaces server implementation, which still has some bugs. Restic creates a lock file in the repo and reads it back, but here the file is not available after creation.

I have some contacts a Digital Ocean and we’re trying to fine the cause for this.

Some workarounds have already been merged to the master branch, but not released yet. If you’d like to try it, we provide compiled versions of the code in master here: https://beta.restic.net/?sort=time&order=desc

3

I just installed 0.8.1 and it seems that the issue is gone. :clap:

I’m just afraid that DO Spaces are not production ready yet… should we switch to something else in your opinion? Suggestions? (We’re trying to avoid big vendors like Amazon)

Many thanks for your time!

4

Hard to say, there’s definitely something odd with Spaces.

5

Hi @fd0 - We are seeing this issue quite frequently occur on our cloud (Openstack + Swiftstack). Quite often, our logging around restic emits this

Load(<lock/69d8b3bf3b>, 0, 0) returned error, retrying after 552.330144ms: The specified key does not exist.
Load(<lock/69d8b3bf3b>, 0, 0) returned error, retrying after 1.080381816s: The specified key does not exist.
Load(<lock/69d8b3bf3b>, 0, 0) returned error, retrying after 1.31013006s: The specified key does not exist.
Load(<lock/69d8b3bf3b>, 0, 0) returned error, retrying after 1.582392691s: The specified key does not exist.
Load(<lock/69d8b3bf3b>, 0, 0) returned error, retrying after 2.340488664s: The specified key does not exist.
Load(<lock/69d8b3bf3b>, 0, 0) returned error, retrying after 4.506218855s: The specified key does not exist.
Load(<lock/69d8b3bf3b>, 0, 0) returned error, retrying after 3.221479586s: The specified key does not exist.
Load(<lock/69d8b3bf3b>, 0, 0) returned error, retrying after 5.608623477s: The specified key does not exist.
Load(<lock/69d8b3bf3b>, 0, 0) returned error, retrying after 7.649837917s: The specified key does not exist.
Load(<lock/69d8b3bf3b>, 0, 0) returned error, retrying after 15.394871241s: The specified key does not exist.

At which point restic bails out with a non-zero code and does not complete backups (or in some other not-so-important cases, does not carry out restic forget ... or restic check ...).

Is there anything we can do to help identify the root cause of this issue on our end ? Thing such as
increasing logging/tracing levels in restic, figuring out which file/key/lock is missing and why, etc.

root@restic-01:~# restic version
restic 0.9.5 compiled with go1.12.4 on linux/amd64