Proxmox backup to synology NAS results in Error using rclone remote

#1

I am new to Restic.
This may not be the correct place to post.
I was using rclone to backup my files from a Proxmox node to a remote location on a Synology NAS.
However, I realized that partial backups could complicate things using rclone alone.
Hence I have been trying to do a backup utilizing rclone as a restic backend.
I mounted the rclone remote using restic and initiated a backup with

Restic -r rclone:remotemis01:Backups/restic/misnode01 backup --verbose /var/lib/vz/dump/vzdump-qemuXXXXXXXX

While the backup is running, I get this repeated message throughout.
Is this okay or is there some real problem going on?

unable to refresh lock: blob not removed, server response: 500 Internal Server Error (500)
[6:48:57] 34.22% 0 files 26.033 GiB, total 1 files 76.075 GiB, 0 errors ETA 13:06:06

#2

Doesn’t look right. Please specify the rclone remotes/backends so we can get a better picture of the configuration and setup.

#3

Thank you for your reply.
The rclone remote is hooked up using SFTP.
Is that the type of information that you are looking for?

#4

Yeah, but can’t you show us the config file for the rclone remote (changing the sensitive information of course)? Maybe there’s nothing in there, it just helps to get a better picture.

#5

Are there any other errors displayed or does restic just repeat that message over and over again? Is restic allowed to create and delete files in the locks folder of the repository? How many files does that folder contain right now (e.g. by counting the lines output by restic list locks)?

A failing lock refresh is problematic if any operation which might remove data from the repository runs concurrently. For example a prune run would normally abort immediately for a locked repository. But if the lock refresh does not work, it could run concurrently and cause data loss if you’re unlucky.

#6

Thanks @MichaelEischer

No other errors, it just keeps repeating that message over and over. I believe that restic is allowed to create and delete files in the locks folder. When I setup the remote, I used the administrator login and password.

I copy and pasted the output into notepad instead of counting out all the lines.
the ouput I got from restic -r (remote) list locks was about 190 lines before I got an error.

rclone: 2020/08/21 18:53:11 ERROR : locks/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxfc53xxxxxxxb1c9a: Delete request remove error: file does not exist
a bunch of times and then
Delete request remove error: file does not exist
error in cleanup handler: blob not removed, server response: 500 Internal Server Error (500)

#7

Hmm, restic list locks should list exactly one lock file (as long as there are no concurrent accesses to the repository). The output should look something like this:

repository d6504c63 opened successfully, password is correct
84a411d2ee44f282814092a4c5b2a20b4a11e34a1e356d519c7fb342b2063093

Can you try whether restic unlock or restic unlock --remove-all helps with your problem? Just make sure that no other restic instance is accessing the repository concurrently.

#8

Hi Michael.
I have run both commands, but it doesn’t seem to resolve the issue. Is there another method to clear out the lock files?

This is the repeated output:

Remove(<lock/001cf890ed>) returned error, retrying after 15.968145272s: blob not removed, server response: 500 Internal Server Error (500)
rclone: 2020/08/22 08:30:49 ERROR : locks/001cf890edea3907897bac3c9c990f648e06e3a6cddf7330e8de4cb9564e84d4: Delete request remove error: file does not exist
blob not removed, server response: 500 Internal Server Error (500)

I had tried to backup another file previously a couple weeks ago. Would this left over also cause issues? I read that it shouldn’t since restic is capable of handling multiple backups and different files.
Is there any way to prune or clean up someof these files. Maybe that would help?
Thank you again.

#9

Prune and every other command which deletes data, requires properly working lock files, so these won’t help. It rather looks like something strange is happening at the backup repository location. Can you manually delete every file in the locks folder of the repository on the NAS? Or is this maybe some permission problem such as not having the permission to delete files?

Which rclone version are you using?

#10

Rclone version output is:
rclone v1.45

  • os/arch: linux/amd64
  • go version: go1.11.6

Is there some special method of deletng the files in the locks folder?
I can do it using the Synology NAS Administrator through the gui, ssh, rclone.
Well anyways I just went through SSH and rm locks/*

Now run a backup normally?

Well I did start it, and it looked like the ETA kept increasing. I could not run it now during operating hours, but I will try again later. The promising thing despite some errors is that there is only one file in the lock folder.

Obviously I got a bunch of erros showing that lock files could not be deleted because they didn’t exist, and then it ended with:

Remove(<lock/8505091614>) returned error, retrying after 16.343935882s: blob not removed, server response: 500 Internal Server Error (500)
rclone: 2020/08/22 09:11:24 ERROR : locks/8505091614f34e07ed0607d2ca58c55859e39371248033031e29743a117f5498: Delete request remove error: file does not exist
Remove(<lock/8505091614>) returned error, retrying after 27.195355545s: blob not removed, server response: 500 Internal Server Error (500)
rclone: 2020/08/22 09:11:52 ERROR : locks/8505091614f34e07ed0607d2ca58c55859e39371248033031e29743a117f5498: Delete request remove error: file does not exist
error in cleanup handler: blob not removed, server response: 500 Internal Server Error (500)

#11

I guess that right now the locks folder is no longer empty? How does rclone connect to the NAS? Via sftp? Could you in that case try the sftp-backend of restic, just to get an better idea where things go wrong?

Are the permissions on the locks folder the same as for the other folder in the repository? You could also remove the locks folder and the recreate it with the same owner/group/permissions as the other folders have.

I have take a look at the rclone changelog and bugtracker, but I haven’t seem something similar to your error messages.

Edit: After deleting all lock files, the locking error should have disappeared.

#12

Thank you so muh Michael. You have been most helpful with this situation.

Well I managed to complete the backup via Rsync since I had already completed something like 40% of the ~75 GiB file.
Also it looked like restic was not advancing steadily anymore. All those errors were causing the process to wait a few milliseconds and then a few seconds and it was stalling the transfer.

I managed to setup SSH with Public Keys and no need for password login between the two machines. Then I just finished setting up SFTP without login just using the Public key.

I have initating a restic utilizing its own sftp-backend. restic -r sftp:user@remote.com:/Storage/restic/mis01 backup —verbose

So far it’s been 3.67 GiB transferred and running for approximately 1 hour without any error messages…
There have been 3 uploaded intermediate indexes.

Is this all a good sign?

I have not tried Ctrl-C and then resuming the transfer yet. I don’t want to mess up the transfer…

#13

I wonder… Can’t you run rest-server on that Synology? It’s way faster than SFTP.

#14

Thanks @rawtaz
I’ll look into it. I did see a post where somebody was able to compile it on a older Synology NAS.
I have access to a DS414, RS816 and a DS918+. So only the DS918+ would be able to run it in Docker. The one I am using right now is the RS816 based on where it is located.

At home I have the DS414, I may try to see if I can get the Rest-server up and running on there before tackling the RS816.

Could you please clarify by way faster? with the sftp-backen right now, I think it is maxing out my bandwidth which is around 10 Mbps upload…

#15

Well the SFTP communication has more overhead than the REST one does, so the latter will make better use of whatever connection you have. Of course there may be other bottlenecks such as disk I/O etc that limit the speed before the transfer protocol does, so your mileage may vary. Try it if you can, if you don’t notice any difference then perhaps it’s not useful to you.

Can’t you build rest-server on your computer? If you use the GOOS and GOARCH variables you can specify the OS and platform to build for, maybe that binary will then run on your NAS.