Slow performance Wasabi S3 bucket

Hello there,

I am attempting to restore one .sql file from a Wasabi S3 hosted bucket, it’s approximately 16GB in size. I tried restoring using the standard restic implementation and now I’m testing using rclone. I previously restored files without any issues but now when I try to restore it takes ages. I left this restore running over last few days and it seems to have only restored around ~8GB of it.

I’m running CentOS 7, with the latest build of restic restic 0.9.6 compiled with go1.13.4 on linux/amd64 (from source). When I try to restore via rclone I get some logs indicating that my restore is going at around 10-25 kBytes/s. Also each time I perform a backup I prune at the end.

Any tips as to what may be causing this dip in performance?

After further lurking the forums for a solution I noticed a comment which mentioned that SMR HDD could impact performance, however this doesn’t seem to be the case for me. I decided it would be worth a shot restoring on a machine with an SSD instead and got the following results:

2020/09/15 14:39:36 DEBUG : S3 bucket example.com: GET /snapshots/xxxxx
2020/09/15 14:39:37 DEBUG : S3 bucket example.com: GET /data/xxxxx1
2020/09/15 14:39:42 INFO  : 
Transferred:   	   31.591M / 31.591 MBytes, 100%, 2.564 MBytes/s, ETA 0s
Errors:                 0
Checks:                 0 / 0, -
Transferred:         9918 / 9918, 100%
Elapsed time:       12.3s

2020/09/15 14:40:42 INFO  : 
Transferred:   	   31.591M / 31.591 MBytes, 100%, 2.564 MBytes/s, ETA 0s
Errors:                 0
Checks:                 0 / 0, -
Transferred:         9918 / 9918, 100%
Elapsed time:       12.3s

2020/09/15 14:41:42 INFO  : 
Transferred:   	   31.591M / 31.591 MBytes, 100%, 2.564 MBytes/s, ETA 0s
Errors:                 0
Checks:                 0 / 0, -
Transferred:         9918 / 9918, 100%
Elapsed time:       12.3s

2020/09/15 14:41:52 DEBUG : S3 bucket example.com: GET /data/xxxxx2
2020/09/15 14:42:42 INFO  : 
Transferred:   	   36.258M / 36.258 MBytes, 100%, 2.387 MBytes/s, ETA 0s
Errors:                 0
Checks:                 0 / 0, -
Transferred:         9919 / 9919, 100%
Elapsed time:       15.1s

2020/09/15 14:43:42 INFO  : 
Transferred:   	   36.258M / 36.258 MBytes, 100%, 2.387 MBytes/s, ETA 0s
Errors:                 0
Checks:                 0 / 0, -
Transferred:         9919 / 9919, 100%
Elapsed time:       15.1s

2020/09/15 14:44:36 DEBUG : S3 bucket example.com: GET /data/xxxxx3
2020/09/15 14:44:42 INFO  : 
Transferred:   	   40.625M / 40.625 MBytes, 100%, 2.367 MBytes/s, ETA 0s
Errors:                 0
Checks:                 0 / 0, -
Transferred:         9920 / 9920, 100%
Elapsed time:       17.1s

Which seems to have slightly improved my performance, but I have a gigabit connection so I was expecting a much better performance. When running ‘top’ it seems that restic is working hard in the background as well with a 100% CPU core usage. Has anyone had a similar sort of issue? My bucket has around 40k+ snapshots.

Maybe you’re affected by the restore perf bug?

I’m not sure if it was related to this, as the speed of my restore was incredibly slow through out the entire restore run while this issue mentions that it progressively slows down. It seems that the fix for this was already merged into master and I tried running a fresh build of master which didn’t seem to resolve the issue. However, when I was lurking yesterday I came across:

And while this issue has already been merged as well it gave me an idea to try the latest beta binary version (just like the person in the issue you mentioned). I used: https://beta.restic.net/restic-v0.9.6-377-gc34c7316/ which seemed to resolve the issue I was having.

Is there a public beta branch for restic? or is that only accessible by devs?

The latest master builds that you already found contains the latest merged restic code, so isn’t that pretty much what you’re asking for here?

But does it also contain any changes that are present on the beta releases? I was slightly confused as I couldn’t figure out what the version on the beta release related to in terms of commits. I went into the commit history of master branch to see if I can find a version relating to:

restic-v0.9.6-377-gc34c7316

But could not find anything relating to this, which lead me to believe there may have been a private Beta repo which gets merged into master, and those version relate directly to the Beta branch.

Do you know by any chance how the versioning there works?

I also compiled the latest master branch along the way and tested my restore using that which seemed to still operate slowly, in hindsight I should’ve deleted any previous version prior to testing, just in case it was still using the old version. I’m gonna try that and report back, may have just been me being an idiot this whole time.

I figured out the versioning and can confirm I’m an idiot. Looks like when performing testing it seems to have used the older version of restic not the fresh one I compiled.

1 Like

It’s simply a direct build of the master branch. So everything that’s been merged will be in it.

If you grab the commit hash from the last part of the filename, but without the initial g, in your case the result would be c34c7316, you can look for that commit hash in the restic repository, e.g. https://github.com/restic/restic/commit/c34c7316 .

I have no idea what that g is about, maybe @fd0 can tell us?