repository 53232826 opened successfully, password is correct
List(lock) returned error, retrying after 552.330144ms: List failed, server response: 404 Not Found (404)
List(lock) returned error, retrying after 1.080381816s: List failed, server response: 404 Not Found (404)
List(lock) returned error, retrying after 1.31013006s: List failed, server response: 404 Not Found (404)
List(lock) returned error, retrying after 1.582392691s: List failed, server response: 404 Not Found (404)
List(lock) returned error, retrying after 2.340488664s: List failed, server response: 404 Not Found (404)
List(lock) returned error, retrying after 4.506218855s: List failed, server response: 404 Not Found (404)
List(lock) returned error, retrying after 3.221479586s: List failed, server response: 404 Not Found (404)
List(lock) returned error, retrying after 5.608623477s: List failed, server response: 404 Not Found (404)
List(lock) returned error, retrying after 7.649837917s: List failed, server response: 404 Not Found (404)
List(lock) returned error, retrying after 15.394871241s: List failed, server response: 404 Not Found (404)
Fatal: unable to create lock in backend: List failed, server response: 404 Not Found (404)
On the server (running rest-server), all repos are missing the locks directory. I checked a few repos, and all are in this fatal state.
Please always include the complete commands and their output, as well as any relevant environment variables, for the rest-server and restic processes you run. This is standard information when describing a problem like this, otherwise you’re just making other people work harder trying to help you. Please also include the versions for both involved software.
Pardon, didn’t come to mind doing that. Had a short moment of panic.
Almost 5h of debugging later, it seems that this is a fault of zfs, nfs and zfs (also with itself) once again fighting due to reasons not fully determined.
I think I’m able to recover from the state and at this point have a lesser reason to believe rest-server or restic is responsible for the original situation.
rest-server seems to be at fault here. Client can be latest Windows or Linux (0.9.6).
Running Docker image da93e5693693 of rest-server. At some point the locking issue has settled in. I could effectively fix a repo by going to the server, giving it the crypt password and running a command such as snapshots via a local: repo. After that was ran, everything seemed to work just fine again.
At this time, I have left few repos in this fatal degraded state, if there’s any information to be gained from there. I’ll need to figure out how to get the --debug output tomorrow. Though I would predict there will not be anything useful. Would need to capture this transformation as it happens.
It should be enough to manually restore the locks using mkdir or something similar, just make sure that the owner an permissions match that of the other folders in the repository.
I don’t see how rest-server should delete the locks folder. There aren’t many code lines which could delete something and these seem to refer only to files.
I think I found the fatal problem. By various software, empty directories are often just ignored and not transferred.
Since restic-server was accessing it through nfs (and me, manually as well), the mkdir’s ownership didn’t match up, as such they were probably 404 due to 403 on the filesystem.
I think the best way to go about this is to create an empty file (.keep) in locks with the restic init command.
If this sounds good, I’ll make plans for a MR.
It’s not reproducible on a regular filesystem. If you can reproduce it, then specify what makes it reproducible.
When I try it, see below, the locks directory is still there after running rclone to rename the repo, and so are the other files and directories. The permissions are fine too.
Regardless, this isn’t a concern of restic’s. If you make a repo, and then something screws up the files restic created in that repo, then obviously the fix needs to be done to whatever trashed the repo’s files. You can’t expect restic to work as it’s designed to work, when some of the files in the repository are not there.
I suggest you init the repo, run a backup, then find -ls it, rclone or whatever it, find -ls it again, and see how it compares. Once you established the symptoms on that part, figure out why your files/dirs are lost or similar.
$ restic version
restic 0.9.6 (v0.9.6-346-geca0f0ad) compiled with go1.14.1 on darwin/amd64
$ rclone version
rclone v1.52.3-DEV
- os/arch: darwin/amd64
- go version: go1.14.1
$ restic init -r local:repo
enter password for new repository:
enter password again:
created restic repository 0da6eaa2ed at local:repo
Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.
$ ls -la
total 0
drwxr-xr-x 3 user staff 96 Aug 10 21:28 .
drwxr-xr-x 37 user staff 1184 Aug 10 21:27 ..
drwx------ 8 user staff 256 Aug 10 21:28 repo
$ find repo -ls
12889581882 0 drwx------ 8 user staff 256 Aug 10 21:28 repo
12889581885 0 drwx------ 2 user staff 64 Aug 10 21:28 repo/locks
12889581883 0 drwx------ 2 user staff 64 Aug 10 21:28 repo/snapshots
12889582145 8 -rw------- 1 user staff 155 Aug 10 21:28 repo/config
12889581884 0 drwx------ 2 user staff 64 Aug 10 21:28 repo/index
12889581886 0 drwx------ 3 user staff 96 Aug 10 21:28 repo/keys
12889582144 8 -rw------- 1 user staff 453 Aug 10 21:28 repo/keys/4e3a0e364b2633f951349c3c94f2e6ca52b677fecc6e421469263c9feeb811fa
12889581887 0 drwx------ 258 user staff 8256 Aug 10 21:28 repo/data
12889581985 0 drwx------ 2 user staff 64 Aug 10 21:28 repo/data/61
12889581901 0 drwx------ 2 user staff 64 Aug 10 21:28 repo/data/0d
12889582037 0 drwx------ 2 user staff 64 Aug 10 21:28 repo/data/95
<snip>
12889582030 0 drwx------ 2 user staff 64 Aug 10 21:28 repo/data/8e
12889581922 0 drwx------ 2 user staff 64 Aug 10 21:28 repo/data/22
12889581925 0 drwx------ 2 user staff 64 Aug 10 21:28 repo/data/25
$ restic -r local:repo backup repo
enter password for repository:
repository 0da6eaa2 opened successfully, password is correct
created new cache in /Users/user/Library/Caches/restic
Files: 3 new, 0 changed, 0 unmodified
Dirs: 0 new, 0 changed, 0 unmodified
Added to the repo: 1.134 KiB
processed 3 files, 780 B in 0:00
snapshot 299b9dda saved
$ rclone move repo repo2
2020/08/10 21:29:53 NOTICE: Config file "/Users/user/.config/rclone/rclone.conf" not found - using defaults
$ ls -la
total 0
drwxr-xr-x 3 user staff 96 Aug 10 21:29 .
drwxr-xr-x 37 user staff 1184 Aug 10 21:27 ..
drwx------ 8 user staff 256 Aug 10 21:28 repo2
$ find repo2 -ls
12889581882 0 drwx------ 8 user staff 256 Aug 10 21:28 repo2
12889581885 0 drwx------ 2 user staff 64 Aug 10 21:29 repo2/locks
12889581883 0 drwx------ 3 user staff 96 Aug 10 21:29 repo2/snapshots
12889582180 8 -rw------- 1 user staff 280 Aug 10 21:29 repo2/snapshots/299b9dda0c09f6c613a4fa3fb091429f729a00f49e58833a1e5598a7b2e6632a
12889582145 8 -rw------- 1 user staff 155 Aug 10 21:28 repo2/config
12889581884 0 drwx------ 3 user staff 96 Aug 10 21:29 repo2/index
12889582177 8 -rw------- 1 user staff 1239 Aug 10 21:29 repo2/index/3c2f2de86a39e9d4df0d103b84513ff3466808f21962f3339e7a9ae95e098b52
12889581886 0 drwx------ 3 user staff 96 Aug 10 21:28 repo2/keys
12889582144 8 -rw------- 1 user staff 453 Aug 10 21:28 repo2/keys/4e3a0e364b2633f951349c3c94f2e6ca52b677fecc6e421469263c9feeb811fa
12889581887 0 drwx------ 258 user staff 8256 Aug 10 21:28 repo2/data
12889581985 0 drwx------ 2 user staff 64 Aug 10 21:28 repo2/data/61
12889581901 0 drwx------ 2 user staff 64 Aug 10 21:28 repo2/data/0d
12889582037 0 drwx------ 2 user staff 64 Aug 10 21:28 repo2/data/95
<snip>
12889582093 0 drwx------ 2 user staff 64 Aug 10 21:28 repo2/data/cd
12889582092 0 drwx------ 2 user staff 64 Aug 10 21:28 repo2/data/cc
12889582118 0 drwx------ 3 user staff 96 Aug 10 21:29 repo2/data/e6
12889582174 192 -rw------- 1 user staff 97841 Aug 10 21:29 repo2/data/e6/e6deaa8859f67d535d4d2c87bddeee4705c48848393d20ae3a7adf110cd35e3b
12889582137 0 drwx------ 2 user staff 64 Aug 10 21:28 repo2/data/f9
12889582128 0 drwx------ 2 user staff 64 Aug 10 21:28 repo2/data/f0
12889582135 0 drwx------ 2 user staff 64 Aug 10 21:28 repo2/data/f7
<snip>
12889581917 0 drwx------ 2 user staff 64 Aug 10 21:28 repo2/data/1d
12889582001 0 drwx------ 2 user staff 64 Aug 10 21:28 repo2/data/71
12889582006 0 drwx------ 3 user staff 96 Aug 10 21:29 repo2/data/76
12889582173 8 -rw------- 1 user staff 1023 Aug 10 21:29 repo2/data/76/76f8b311ce1dc718df4fe182c0d8816267a57361f0b497494f803f8b37d8be43
12889581916 0 drwx------ 2 user staff 64 Aug 10 21:28 repo2/data/1c
12889582018 0 drwx------ 2 user staff 64 Aug 10 21:28 repo2/data/82
12889581961 0 drwx------ 2 user staff 64 Aug 10 21:28 repo2/data/49
<snip>
12889582030 0 drwx------ 2 user staff 64 Aug 10 21:28 repo2/data/8e
12889581922 0 drwx------ 2 user staff 64 Aug 10 21:28 repo2/data/22
12889581925 0 drwx------ 2 user staff 64 Aug 10 21:28 repo2/data/25
$ restic -r local:repo2 snapshots
enter password for repository:
repository 0da6eaa2 opened successfully, password is correct
ID Time Host Tags Paths
--------------------------------------------------------------------------------------------------------
299b9dda 2020-08-10 21:29:30 host.local /Users/user/test/repo
--------------------------------------------------------------------------------------------------------
1 snapshots