Errors 500 with rclone and google drive

Getting Help
1

I’m trying restic for a fairly large backup (~800G) directly to a drive account. My versions are:

$ restic version
restic 0.9.4 compiled with go1.11.4 on linux/amd64
$ rclone version
rclone v1.46
- os/arch: linux/amd64
- go version: go1.11.5

rclone is configured to use 2 transfers and 4 checkers.

During backup, I see sporadic 403 errors from rclone which I believe are benign. However, I see too a few 500 errors:

Save(<index/d8f59d1e23>) returned error, retrying after 598.926326ms: server response unexpected: 500 Internal Server Error (500)

After the backup is complete, restic check fails. Trying to salvage the upload with rebuild-index has failed in a couple of attempts, because the errors 500 also happen while loading the indexes.

Any ideas on how to diagnose those errors?

1 Like
2

I’ve seen a lot of errors (I don’t really remember if 500 is one of them) using Google Drive. I think almost all errors are due to Drive restrictions to upload but if you let the process continue it will be completed at some point. I did a test with 15GiB with my Google Drive account and it took almost 15 hours to complete backup, forget, prune and check. I haven’t tried this myself but you could try to use Google API directly as suggested here.

3

I will try that, thanks.

As an update, even lowering both checkers and transfers to 1, I see 403 and 500 happen from time to time.

4

To be honest, I would recommend avoiding Google Drive as a backup solution because of this issue and others that users on this forum have encountered. Drive is designed to be used primarily by people and not scripts, and there are restrictions based on this assumption.

Another way to look at it: If you’re having these problems now, would you want to risk encountering them when you urgently need to restore a backup?

I’d suggest having a look at B2. Storage pricing is about half of Google Drive’s pricing and requires no commitment (you pay only for what you use each month), with the caveat that downloading data from B2 is not free (but still quite reasonable, especially if restores are rare).

Google Drive pricing for 1TB is $9.99/mo. Storing 1TB in B2 would incur $5.12/mo of charges, but if you store less in a given month then you pay less.

1 Like
5

According to rclone’s documentation on Google Drive uploads are limited to roughly 2 files per second. I doubt that that tweaking the settings will help a lot in this case.

6

It turns out that the problem is not checkers/transfers, but requests per second. After creating my own client id and adding --tpslimit=10 I see no more errors, either 403 or 500.

There’s still the problem that two concurrent rclone runs will overstep the limit, but being careful with my cron jobs, this should do it for my needs.

8

Dang. “unknown flag: --tpslimit” on 2022-09-12 (command “backup”). I have already included --limit-upload and --limit-download. Can’t see anything which covers “requests per second”.

However, I do see --option key=value “set extended option”.

Is there still a way to limit “TPS”?

Full details of final output before killing itself:

rclone: 2022/09/12 00:29:53 ERROR : data/5e/5eccd44350719651de5fc9c96d67857b428508478503f1a9f07ea2e241072ee7:
Post request rcat error: context canceled
rclone: 2022/09/12 00:29:53 ERROR : data/19/19ec6c5ba31ace49e9d3bbc4fc7bf803c38995b919065bfb186659d829d934fc:
Post request put error: couldn’t list directory: context canceled
rclone: 2022/09/12 00:29:53 ERROR : data/19/19ec6c5ba31ace49e9d3bbc4fc7bf803c38995b919065bfb186659d829d934fc:
Post request rcat error: couldn’t list directory: context canceled
Fatal: unable to save snapshot: server response unexpected: 500 Internal Server Error (500)

Tried again: same errors, ending with same “Fatal”.

Also seen this and the fix. My version is 0.12.0 so seems it will incorporate that fix.

By the way, I’m doing this once a week on a 5 GB set of directories. It worked last week and the week before. Both times many errors were reported but it finally finished, and restore testing showed that it was completing OK.

9

Dang. “unknown flag: --tpslimit” on 2022-09-12 (command “backup”). I have already included --limit-upload and --limit-download. Can’t see anything which covers “requests per second”.

However, I do see --option key=value “set extended option”.

Is there still a way to limit “TPS”?

the TPS limit (and checkers/transfers) are all set with rclone, not restic
you could set the TPS limit as an rclone environmental variable with
export RCLONE_TPSLIMIT=1
on the command line for example.

Or as an rclone argument:
--tpslimit 1 (for 1 transfer/second)
which can be passed to rclone using restic with the restic argument:
-o rclone.args
for example using:
restic -o rclone.args="serve restic --stdio --bwlimit 1M --b2-hard-delete --verbose --tpslimit 1"
(see https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#other-services-via-rclone)

See the main rclone arguments documentation page:
https://rclone.org/docs/
where you can check –tpslimit, –checkers, and –transfers

10

you could also try lowering the concurrent transfers directly from restic, that seems to work really well for me:
restic backup -o rclone.connections=1
(see https://restic.readthedocs.io/en/stable/047_tuning_backup_parameters.html#backend-connections)