Errors 500 with rclone and google drive

#1

I’m trying restic for a fairly large backup (~800G) directly to a drive account. My versions are:

$ restic version
restic 0.9.4 compiled with go1.11.4 on linux/amd64
$ rclone version
rclone v1.46
- os/arch: linux/amd64
- go version: go1.11.5

rclone is configured to use 2 transfers and 4 checkers.

During backup, I see sporadic 403 errors from rclone which I believe are benign. However, I see too a few 500 errors:

Save(<index/d8f59d1e23>) returned error, retrying after 598.926326ms: server response unexpected: 500 Internal Server Error (500)

After the backup is complete, restic check fails. Trying to salvage the upload with rebuild-index has failed in a couple of attempts, because the errors 500 also happen while loading the indexes.

Any ideas on how to diagnose those errors?

Backup process doesn't work anymore, stuck a locking the repository
#2

I’ve seen a lot of errors (I don’t really remember if 500 is one of them) using Google Drive. I think almost all errors are due to Drive restrictions to upload but if you let the process continue it will be completed at some point. I did a test with 15GiB with my Google Drive account and it took almost 15 hours to complete backup, forget, prune and check. I haven’t tried this myself but you could try to use Google API directly as suggested here.

#3

I will try that, thanks.

As an update, even lowering both checkers and transfers to 1, I see 403 and 500 happen from time to time.

#4

To be honest, I would recommend avoiding Google Drive as a backup solution because of this issue and others that users on this forum have encountered. Drive is designed to be used primarily by people and not scripts, and there are restrictions based on this assumption.

Another way to look at it: If you’re having these problems now, would you want to risk encountering them when you urgently need to restore a backup?

I’d suggest having a look at B2. Storage pricing is about half of Google Drive’s pricing and requires no commitment (you pay only for what you use each month), with the caveat that downloading data from B2 is not free (but still quite reasonable, especially if restores are rare).

Google Drive pricing for 1TB is $9.99/mo. Storing 1TB in B2 would incur $5.12/mo of charges, but if you store less in a given month then you pay less.

1 Like
#5

According to rclone’s documentation on Google Drive uploads are limited to roughly 2 files per second. I doubt that that tweaking the settings will help a lot in this case.

Backup process doesn't work anymore, stuck a locking the repository
#6

It turns out that the problem is not checkers/transfers, but requests per second. After creating my own client id and adding --tpslimit=10 I see no more errors, either 403 or 500.

There’s still the problem that two concurrent rclone runs will overstep the limit, but being careful with my cron jobs, this should do it for my needs.

#7

After the install itunes on windows 7 I have checked that the cloud service was not working at all. I have even tried to run the application with admin privileges but still it was not working so please help me to fix it.