JohKa
May 2, 2022, 7:14am
1
Good Morning everybody,
I updated my restic to the latest version when it came out, since then I get the following errors with every command:
open repository
lock repository
Save(<lock/44931584af>) returned error, retrying after 552.330144ms: open /mnt/backup/locks/44931584af7bfb19ff8c7b2652c3ae1b0ef17f4bc3c4bb6165cf4bdf11610313-tmp-3781515316: operation not supported
Even the restic -r /mnt/backup check
command fails with the error above.
If I try to run the command with the flag --no-lock
it works.
Any body know how to fix this?
Regards,
fd0
May 2, 2022, 7:16am
2
First, you need to tell us:
Which restic version do you use currently (run restic version
)?
Which was the latest restic version that worked?
What is mounted at /mnt/backup
? Which filesystem does it use? (run findmnt /mnt/backup
and paste the output)
JohKa
May 2, 2022, 7:25am
3
Hey,
thanks for your quick reply:
Here are the infos you asked for:
Version: restic 0.13.1 compiled with go1.18 on linux/amd64
Previous Version: restic 0.12.1
findmnt output: /mnt/backup curlftpfs#ftp://***:***@backup.***.**/ fuse rw,nosuid,nodev,relatime,user_id=0,group_id=0
fd0
May 2, 2022, 11:37am
4
I suspect that this is caused by curlftpfs
not supporting the sync()
syscall, which should ensure that data is really written to the backend. A similar issue is documented here:
opened 02:17AM - 24 Apr 22 UTC
type: bug
Output of `restic version`
--------------------------
> restic version
restic… 0.13.1 compiled with go1.18 on darwin/arm64
~/.restic
How did you run restic exactly?
-------------------------------
from a script file with contents of:
```
#! /bin/bash -e
export RESTIC_REPOSITORY=/Volumes/restic-jeff/MBPr15
export RESTIC_PASSWORD= <<REDACTED>>
./restic_0.12.1_darwin_arm64 --verbose --tag MacBookAir2020 backup ~/Dendron
```
What backend/server/service did you use to store the repository?
----------------------------------------------------------------
SMB
Expected behavior
-----------------
Not to error out creating lock file
Actual behavior
---------------
This is the results of running the script
(note that if I watch that folder in Finder, I do see a lock file beginning with that name appear and then disappear in the locks directory)
```
> ./backup-dendron.sh
open repository
repository 69a19a6a opened successfully, password is correct
lock repository
Save(<lock/071fe833f0>) returned error, retrying after 552.330144ms: sync /Volumes/restic-jeff/MBPr15/locks: no such file or directory
Save(<lock/071fe833f0>) returned error, retrying after 1.080381816s: sync /Volumes/restic-jeff/MBPr15/locks: no such file or directory
Save(<lock/071fe833f0>) returned error, retrying after 1.31013006s: sync /Volumes/restic-jeff/MBPr15/locks: no such file or directory
Save(<lock/071fe833f0>) returned error, retrying after 1.582392691s: sync /Volumes/restic-jeff/MBPr15/locks: no such file or directory
Save(<lock/071fe833f0>) returned error, retrying after 2.340488664s: sync /Volumes/restic-jeff/MBPr15/locks: no such file or directory
Save(<lock/071fe833f0>) returned error, retrying after 4.506218855s: sync /Volumes/restic-jeff/MBPr15/locks: no such file or directory
Save(<lock/071fe833f0>) returned error, retrying after 3.221479586s: sync /Volumes/restic-jeff/MBPr15/locks: no such file or directory
Save(<lock/071fe833f0>) returned error, retrying after 5.608623477s: sync /Volumes/restic-jeff/MBPr15/locks: no such file or directory
Save(<lock/071fe833f0>) returned error, retrying after 7.649837917s: sync /Volumes/restic-jeff/MBPr15/locks: no such file or directory
Save(<lock/071fe833f0>) returned error, retrying after 15.394871241s: sync /Volumes/restic-jeff/MBPr15/locks: no such file or directory
sync /Volumes/restic-jeff/MBPr15/locks: no such file or directory
github.com/restic/restic/internal/backend/local.(*Local).Save
github.com/restic/restic/internal/backend/local/local.go:169
github.com/restic/restic/internal/limiter.rateLimitedBackend.Save
github.com/restic/restic/internal/limiter/limiter_backend.go:30
github.com/restic/restic/internal/backend.(*RetryBackend).Save.func1
github.com/restic/restic/internal/backend/backend_retry.go:66
github.com/cenkalti/backoff/v4.RetryNotifyWithTimer
github.com/cenkalti/backoff/v4@v4.1.1/retry.go:55
github.com/cenkalti/backoff/v4.RetryNotify
github.com/cenkalti/backoff/v4@v4.1.1/retry.go:34
github.com/restic/restic/internal/backend.(*RetryBackend).retry
github.com/restic/restic/internal/backend/backend_retry.go:46
github.com/restic/restic/internal/backend.(*RetryBackend).Save
github.com/restic/restic/internal/backend/backend_retry.go:60
github.com/restic/restic/internal/cache.(*Backend).Save
github.com/restic/restic/internal/cache/backend.go:59
github.com/restic/restic/internal/repository.(*Repository).SaveUnpacked
github.com/restic/restic/internal/repository/repository.go:330
github.com/restic/restic/internal/repository.(*Repository).SaveJSONUnpacked
github.com/restic/restic/internal/repository/repository.go:310
github.com/restic/restic/internal/restic.(*Lock).createLock
github.com/restic/restic/internal/restic/lock.go:163
github.com/restic/restic/internal/restic.newLock
github.com/restic/restic/internal/restic/lock.go:107
github.com/restic/restic/internal/restic.NewLock
github.com/restic/restic/internal/restic/lock.go:68
main.lockRepository
github.com/restic/restic/cmd/restic/lock.go:42
main.lockRepo
github.com/restic/restic/cmd/restic/lock.go:23
main.runBackup
github.com/restic/restic/cmd/restic/cmd_backup.go:556
main.glob..func2
github.com/restic/restic/cmd/restic/cmd_backup.go:61
github.com/spf13/cobra.(*Command).execute
github.com/spf13/cobra@v1.2.1/command.go:856
github.com/spf13/cobra.(*Command).ExecuteC
github.com/spf13/cobra@v1.2.1/command.go:974
github.com/spf13/cobra.(*Command).Execute
github.com/spf13/cobra@v1.2.1/command.go:902
main.main
github.com/restic/restic/cmd/restic/main.go:98
runtime.main
runtime/proc.go:250
runtime.goexit
runtime/asm_arm64.s:1259
unable to create lock in backend
>
```
Steps to reproduce the behavior
-------------------------------
Run Restic with any command that requires creating a lock file
Do you have any idea what may have caused this?
-----------------------------------------------
Something in v0.13.0 seems to have caused this.
I get this error with both the 0.13.0 and 0.13.1 binaries, but when I run with the 0.12.1 binary it works fine.
The symptom seems to be very similar to a prior issue: https://github.com/restic/restic/issues/2395
Do you have an idea how to solve the issue?
-------------------------------------------
Unfortunately, no. For now I've reverted back to running v0.12.1
Did restic help you today? Did it make you happy in any way?
------------------------------------------------------------
Restic has been an awesome backup tool...but something in v13 broke it for me :(
We have a list of cases in which restic just ignores the error returned by sync()
, and it seems we need to extend this list quite a bit.
Are you able to build restic yourself? Can you please build and test it with the patch from the issue ?
JohKa
May 2, 2022, 1:09pm
5
I’ll try this in a few days. Thank you.
1 Like
Errhh, that error is not returned by sync. It is returned by open
and that is only called within ioutil.TempFile
using O_RDWR|O_CREATE|O_EXCL
as flags. Previously, in restic 0.12.1 os.O_CREATE|os.O_EXCL|os.O_WRONLY
was used. So unless we want to reimplement the temp file creation there’s not much we can fix here.
@JohKa is there a particular reason to use curlftpfs instead of using rclone to access the ftp repository?
fd0
May 11, 2022, 6:42pm
7
Oh, yeah, you’re right! Sorry about the confusion.
JohKa
June 8, 2022, 6:53am
8
I’m sorry to answer so late.
I was not able to build restic by my own. So I tried to use rclone for mounting instead.
But every time I get the following error which seems to be rclone related but maybe you have an idea:
Save(<data/a06ed4ce05>) returned error, retrying after 552.330144ms: open /mnt/backup/data/a0/a06ed4ce05b52471ba21a7d32504987165ff0edc33436411c39e03b8931f2e66-tmp-1134479176: input/output error
What was the problem with building restic?
That error looks like you’re not using rclone at all but rather the local
backend. What is the exact command line you’re using?
JohKa
June 12, 2022, 10:11am
10
It ist just a time problem no functional problem.
# /etc/systemd/system/rclone-backup.service
[Unit]
Description=Backup (rclone)
AssertPathIsDirectory=/mnt/backup
#After=plexdrive.service
[Service]
Type=simple
ExecStart=/usr/bin/rclone mount \
--config=/root/.config/rclone/rclone.conf \
--allow-other \
--dir-perms 0777 \
# --file-perms 0666 \
# --cache-tmp-upload-path=/tmp/rclone/upload \
# --cache-chunk-path=/tmp/rclone/chunks \
# --cache-workers=8 \
# --cache-writes \
# --cache-dir=/tmp/rclone/vfs \
# --cache-db-path=/tmp/rclone/db \
# --no-modtime \
# --drive-use-trash \
# --stats=0 \
# --checkers=16 \
# --bwlimit=40M \
# --dir-cache-time=60m \
# --cache-info-age=60m \
-vvv \
"Backupspace": /mnt/backup/
ExecStop=/bin/fusermount -u /mnt/backup
Restart=always
RestartSec=10
[Install]
WantedBy=default.target
I also tried it with the commented out configs but it changed nothing in the behaviour.
and the rclone Backend configuration:
[Backupspace]
type = ftp
host = backup.XXXX.net
user = XXX
pass = XXX
I’d strongly recommend to let restic interact with rclone directly without the indirection of a fuse mount:
restic -o rclone.args="serve restic --stdio --b2-hard-delete --config=/root/.config/rclone/rclone.conf" -r rclone:Backupspace: [...]
JohKa
June 12, 2022, 10:41am
12
I tried this in my script with:
RESTIC_ARGS='-o rclone.args="serve restic --stdio --b2-hard-delete --config=/root/.config/rclone/rclone.conf" -r rclone:Backupspace:'
and commands like this
restic $RESTIC_ARGS backup --verbose -one-file-system --tag "mail" --tag "files" /var/vmail/
but I’m getting the following error:
unknown command "restic" for "restic"
And with executing the command directly in the shell I get the following error:
ERROR : data/9a/9a346f9fa177a9174fcde91382c11c4ec6f14780160384fe82953d8e55637b45: Post request put error: Put mkParentDir failed: mkdir "data/9a" failed: EOF
rclone: 2022/06/12 13:08:54 ERROR : data: error listing: EOF
rclone: 2022/06/12 13:08:54 ERROR : data: list failed: &errors.errorString{s:"EOF"} *errors.errorString
List(data) returned error, retrying after 720.254544ms: List failed, server response: 404 Not Found (404)
rclone: 2022/06/12 13:08:55 ERROR : data: error listing: EOF
rclone: 2022/06/12 13:08:55 ERROR : data: list failed: &errors.errorString{s:"EOF"} *errors.errorString
List(data) returned error, retrying after 873.42004ms: List failed, server response: 404 Not Found (404)
List(data) returned error, retrying after 1.054928461s: List failed, server response: 404 Not Found (404)
rclone: 2022/06/12 13:08:55 ERROR : data: error listing: EOF
rclone: 2022/06/12 13:08:55 ERROR : data: list failed: &errors.errorString{s:"EOF"} *errors.errorString
rclone: 2022/06/12 13:08:56 ERROR : data: error listing: EOF
rclone: 2022/06/12 13:08:56 ERROR : data: list failed: &errors.errorString{s:"EOF"} *errors.errorString
List(data) returned error, retrying after 1.560325776s: List failed, server response: 404 Not Found (404)
List(data) returned error, retrying after 3.004145903s: List failed, server response: 404 Not Found (404)
rclone: 2022/06/12 13:08:58 ERROR : data: error listing: EOF
rclone: 2022/06/12 13:08:58 ERROR : data: list failed: &errors.errorString{s:"EOF"} *errors.errorString
rclone: 2022/06/12 13:14:08 ERROR : locks/b6c9370156c9425e0dbf817440a4f41ba9bfc16d96c9661f1a9105ebe57ab280: Post request put error: Put mkParentDir failed: mkdir "locks" failed: write tcp 5.189.139.153:47120->213.136.95.26:21: write: broken pipe
rclone: 2022/06/12 13:14:08 ERROR : locks/b6c9370156c9425e0dbf817440a4f41ba9bfc16d96c9661f1a9105ebe57ab280: Post request rcat error: Put mkParentDir failed: mkdir "locks" failed: write tcp 5.189.139.153:47120->213.136.95.26:21: write: broken pipe
Save(<lock/b6c9370156>) returned error, retrying after 720.254544ms: server response unexpected: 500 Internal Server Error (500)
That means that the quoting for $RESTIC_ARGS is incorrect. RESTIC_ARGS='-o rclone.args=serve\ restic\ --stdio\ --b2-hard-delete\ --config=/root/.config/rclone/rclone.conf -r rclone:Backupspace:'
might work.
The other error log looks like rclone failed somewhere before the log excerpt.