I stopped a backup in progress by accident and now the lock won’t remove. I had this before and I did what I always did:
restic -r b2:repo/folder unlock
restic -r b2:repo/folder rebuild-index
restic -r b2:repo/folder prune
restic -r b2:repo/folder check
But I did not work this time. I still get the following every time I try to back up:
Fatal: unable to create lock in backend: repository is already locked by PID 12560 on host by user (UID 1000, GID 1000)
What else can I do?
What’s the complete output from the unlock command when you run it?
You really shouldn’t need anything but run the unlock command, assuming there’s no other (restic) process using the repo. This should remove the lock files.
If it doesn’t, I would suspect eventual consistency of B2. It’s not the first time we’ve seen what indicates that from B2, it seems to tell you one thing then another the next second 
Thanks for the quick reply! I see…
This is funny. This is what I get when I run the backup command directly from the command line:
source /path/to/creds.txt
restic -r b2:bucket:/path backup --verbose /path/to/source --password-file /path/to/pwfile.txt --exclude="**/Temp-" --limit-upload 2500 > /path/to/xxxxxx.log
open repository
repository xxxxxx opened successfully, password is correct
lock repository
load index files
using parent snapshot xxxxxx
start scan on [/media/xxxxxx]
start backup on [/media/xxxxxx]
scan finished in 11.677s: 92134 files, 503.786 GiB
Files: 0 new, 8 changed, 92126 unmodified
Dirs: 0 new, 1 changed, 0 unmodified
Data Blobs: 11 new
Tree Blobs: 2 new
Added to the repo: 8.375 MiB
processed 92134 files, 503.786 GiB in 0:33
snapshot xxxxxx saved
When I run the batch file from the command line, which contains the exact same backup command (but parses output to a file and file gets mailed) I get the same, usual output in my mail but where the output in cli should be blank it gives this again, this happens whit ‘restic forget’:
source /path/to/creds.txt
restic forget --repo b2:bucket:/path --password-file /path/to/pwfile.txt --keep-daily 21 --keep-weekly 14 --keep-monthly 36 > /path/to/xxxxxx.log
Fatal: unable to create lock in backend: repository is already locked by PID 12560 on host by user (UID 1000, GID 1000)
lock was created at 2020-04-19 08:30:06 (30h43m42.455048354s ago)
storage ID xxxxxx
mail: Null message body; hope that's okIt’s much easier for us to follow your context if you include the complete commands that you run (and in case of scripts, the scripts). Can you please click the edit button on your post and update it with that information?
The lock file about which the batch file run complains is old enough to be deleted by running unlock. Since it’s 30 hours old, there shouldn’t be any visibility problems regarding that file in B2 (at least it’s very unlikely).
Could you run restic list locks and check whether any of those files begins with the id printed after storage ID?
The problematic lock cannot be the left-over of an aborted backup run as restic allows multiple backup run in parallel and these therefore wouldn’t conflict. The lock file must be the remainder of a rebuild-index/prune/check run.
I did that. It happens when I run ‘restic forget’.
Thank you. It won’t go away with ‘restic unlock’ but ‘restic list locks’ shows this:
repository xxxxxx opened successfully, password is correct
**3ce24d61**1ecc4eda4b4344b202bbe4c902c74218405153da1aad72f3069b1f83
5f634cff6e6e0670e823ac670d7f3bfe1e35a146144d625144599ec866cd65fe
b902c20beafffb386be372df0e6a0c763fea9b81772b661a49ee98c3c4597174
Marked with ** is indeed the storage ID
Can you check using restic cat lock <id> whether restic is able to read the lock file? Which value is stored in the time field of the lock?
What is the output of restic unlock -v? The unlock command should remove every lock file that is older than 30 minutes, which is apparently the case for your lock file.
Thank you for your reply. The lock somehow disappeared? I did run the commands. I guess the output is normal since the lock is somehow gone?
restic cat lock 3ce24d611ecc4eda4b4344b202bbe4c902c74218405153da1aad72f3069b1f83 -r b2:bucket:/DATA --password-file pwfile.txt
repository 87a32f17 opened successfully, password is correct
Load(<lock/3ce24d611e>, 0, 0) returned error, retrying after 603.421495ms: b2_download_file_by_name: 404: bucket bucket does not have file: DATA/locks/3ce24d611ecc4eda4b4344b202bbe4c902c74218405153da1aad72f3069b1f83
Load(<lock/3ce24d611e>, 0, 0) returned error, retrying after 1.030706132s: b2_download_file_by_name: 404: bucket bucket does not have file: DATA/locks/3ce24d611ecc4eda4b4344b202bbe4c902c74218405153da1aad72f3069b1f83
Load(<lock/3ce24d611e>, 0, 0) returned error, retrying after 1.123022702s: b2_download_file_by_name: 404: bucket bucket does not have file: DATA/locks/3ce24d611ecc4eda4b4344b202bbe4c902c74218405153da1aad72f3069b1f83
Load(<lock/3ce24d611e>, 0, 0) returned error, retrying after 1.861579714s: b2_download_file_by_name: 404: bucket bucket does not have file: DATA/locks/3ce24d611ecc4eda4b4344b202bbe4c902c74218405153da1aad72f3069b1f83
Load(<lock/3ce24d611e>, 0, 0) returned error, retrying after 2.8293048s: b2_download_file_by_name: 404: bucket bucket does not have file: DATA/locks/3ce24d611ecc4eda4b4344b202bbe4c902c74218405153da1aad72f3069b1f83
Load(<lock/3ce24d611e>, 0, 0) returned error, retrying after 2.175139449s: b2_download_file_by_name: 404: bucket bucket does not have file: DATA/locks/3ce24d611ecc4eda4b4344b202bbe4c902c74218405153da1aad72f3069b1f83
Load(<lock/3ce24d611e>, 0, 0) returned error, retrying after 4.76025921s: b2_download_file_by_name: 404: bucket bucket does not have file: DATA/locks/3ce24d611ecc4eda4b4344b202bbe4c902c74218405153da1aad72f3069b1f83
Load(<lock/3ce24d611e>, 0, 0) returned error, retrying after 10.980108495s: b2_download_file_by_name: 404: bucket bucket does not have file: DATA/locks/3ce24d611ecc4eda4b4344b202bbe4c902c74218405153da1aad72f3069b1f83
Load(<lock/3ce24d611e>, 0, 0) returned error, retrying after 8.448958083s: b2_download_file_by_name: 404: bucket bucket does not have file: DATA/locks/3ce24d611ecc4eda4b4344b202bbe4c902c74218405153da1aad72f3069b1f83
Load(<lock/3ce24d611e>, 0, 0) returned error, retrying after 23.247318569s: b2_download_file_by_name: 404: bucket bucket does not have file: DATA/locks/3ce24d611ecc4eda4b4344b202bbe4c902c74218405153da1aad72f3069b1f83
b2_download_file_by_name: 404: bucket bucket does not have file: DATA/locks/3ce24d611ecc4eda4b4344b202bbe4c902c74218405153da1aad72f3069b1f83
and:
restic unlock -v
enter password for repository:
repository 87a32f17 opened successfully, password is correct
successfully removed locksIf the lock has disappeared (can you check again with restic list locks?) then the 404 not found errors from b2 are expected.
Hm, that’s not the case
restic list locks -r b2:bucket:/DATA --password-file pwfile.txt
repository xxxxxx opened successfully, password is correct
e6c621a16ba175c22b2c75f0b4f4d68a7b7d63c45812eae8689643700029bd9cThe list command also creates a lock file unless you specify --no-lock. So the single listed lock file is the one created by list.
Ok. I guess it is solved then. Thank you!