Performance of transfer (time taken) for restic backup vs rclone copy

Getting Help
1

Good day!

intro

I’m trying to backup sites (wordpress mostly, so small files per each site) and working with storage boxes via rclone (rclone helps abstract different storage engines be it SFTP/S3 AWS/S3 WASABI and all managed in single config file).

So, rclone has remote named wasabis3-sites-backup and I configure restic via environment var RESTIC_REPOSITORY=rclone:wasabis3-sites-backup:

Storage: Wasabi S3, in close proximity, ~ 600km , not over the globe
Restic ( 0.12.1) and Rclone ( v1.57.0) - latest ones

issues

Then i’ve tried simple “backup” command for restic and

it’s very slow uploading to the remote, just 4k files, 64MB has taken near 5 minutes

Files:        3481 new,     0 changed,     0 unmodified
Dirs:          517 new,     0 changed,     0 unmodified
Data Blobs:    679 new
Tree Blobs:    514 new
Added to the repo: 18.631 MiB

processed 3481 files, 64.119 MiB in 4:41

the same site, the same remote, via rclone copy

2021/12/15 23:26:32 INFO  :  
Transferred:       21.117 MiB / 21.117 MiB, 100%, 31.234 KiB/s, ETA 0s                                                                                                                                                                      Checks:              2249 / 2249, 100% 
Transferred:         1076 / 1076, 100%
Elapsed time:       2m4.1s

This looks very strange for me - to my understanding restic backup should pack multiple small files in one larger blob (4MB each) and upload those blobs, which for S3 type storage should have much more advantages over file-per-file copy as rclone copy does

a lot of errors arise when using restic over rclone for this case

error messages like

rclone: 2021/12/15 23:39:58 ERROR : data/61/61f47e58a4a3183cbb5daaeee555ddf2ef6632813f5c480e3cab292b7d414089: Post request put error: s3 upload: 400 Bad Request: <?xml version="1.0" encoding="UTF-8"?>
rclone: <Error><Code>RequestTimeout</Code><Message>Your socket connection to the server was not read from or written to within the timeout period, source:  read tcp 130.117.252.20:443-&gt;157.x.x.x:33184: i/o timeout</Message><RequestId>76DF166B220679D0</RequestId><HostId>QNuh4MnKcTNEZVIxv26iBex/6KvTI3JLe/r7Vkj1w87MFRhLxQ47OKHHDdO89zIuFBB57oDAIFfW</HostId></Error>
rclone: 2021/12/15 23:40:07 ERROR : data/61/61f47e58a4a3183cbb5daaeee555ddf2ef6632813f5c480e3cab292b7d414089: Post request rcat error: s3 upload: 400 Bad Request: <?xml version="1.0" encoding="UTF-8"?>
rclone: <Error><Code>RequestTimeout</Code><Message>Your socket connection to the server was not read from or written to within the timeout period, source:  read tcp 130.117.252.20:443-&gt;157.x.x.x:33184: i/o timeout</Message><RequestId>76DF166B220679D0</RequestId><HostId>QNuh4MnKcTNEZVIxv26iBex/6KvTI3JLe/r7Vkj1w87MFRhLxQ47OKHHDdO89zIuFBB57oDAIFfW</HostId></Error>
Save(<data/61f47e58a4>) returned error, retrying after 1.560325776s: server response unexpected: 500 Internal Server Error (500)
...
...
...
Save(<data/d92b1d9e10>) returned error, retrying after 593.411537ms: server response unexpected: 500 Internal Server Error (500)
Save(<data/6d712458c6>) returned error, retrying after 282.818509ms: server response unexpected: 500 Internal Server Error (500)
Save(<data/30993bcc8f>) returned error, retrying after 328.259627ms: server response unexpected: 500 Internal Server Error (500)
Save(<data/df5d1b8584>) returned error, retrying after 298.484759ms: server response unexpected: 500 Internal Server Error (500)
Save(<data/1c2dc7fe90>) returned error, retrying after 400.45593ms: server response unexpected: 500 Internal Server Error (500)

and so on.

May be I’m doing something wrong, please advise.

2

IIRC I have read many times that wasabi storage can be a bit flaky… have you searched the forum for wasabi related issues? You are probably not the only one.

But what would be interesting to know if the problem is unrelated to restic at all.
Can you use rclone to put some data to the wasabi bucket and see if the problem occurs there as well?

3

using rclone copy as in my original message or by some other way?

4

Ah, sorry! I totally missed that you tried it with rclone directly as well.
I think what you need in that case are some tweaking options to configure timeouts for the Wasabi S3 backend.

As it’s the Christmas holidays I don’t have the time right now to dig in and find the options but I guess if you either google or search the forums for “restic rclone wasabi” you will find options to tweak the connection - especially the timeouts.

5

On a bright side, now speed is fine - seems it was related by network maintenace made by my hoster (Hetzner) during that time.

1 Like
6

Well I’m glad it resolved itself. Sometimes local network issues are a thing and I keep forgetting about it :slight_smile: I’ll mark this thread as solved then.

1 Like