S3 Bottleneck vs Rest Server

We are searching for a Borgbackup alternative that can help us achieve faster backups. With borg, we are limited by the single connection bandwidth limit.

While Restic uses concurrent uploads we are getting poor performance from S3 vs Rest Server.

With S3 we are looking at around 800M for the initial backup and 200-400M for incremental backups.
For Restic Server we are always in the > 2G space.

Both S3 (Minio) and Rest Server are on the same machine. With iperf3 the speed is > 8G between them.

We are getting way better Minio performance with other applications.
Does someone know what the S3 implementation bottleneck is and if what we are seeing is normal?
Is there anything that can be tuned to get S3 into the G space?

Thanks

There’s no such thing as “Restic Server”. You are probably referring to REST Server, which is made to be used with restic.

Yes this one.

Just to check in, what is your restic version? 0.9.x was noticeably worse than the latest ones in regards to s3.

We tested both 0.13.1 as well as a custom build of this pull request Adjust worker goroutines to number of backend connections by MichaelEischer · Pull Request #3611 · restic/restic · GitHub

Any thoughts on this huge performance difference?

Not much, you can try different -o s3.connections=N values to check if this will bring an improvement. But I am not sure if that’s essentially same thing of the PR linked above.

Does using a larger pack size via Feature: min packsize flag by metalsp0rk · Pull Request #3731 · restic/restic · GitHub improve throughput?

Does maybe MinIO or rest-server use http/1.1 and the other one http/2? Does setting GODEBUG=http2client=0 as environment variable for restic change anything?

For Minio setting this to 20 brought it to 3G. For the external S3 increasing it did not make a difference.

1 Like

changing this did not make a difference

My guess right now would be that MinIO takes more time to store uploaded data than the rest-server. At least that would be consistent with the reported numbers. Although I’m not completely sure how the external S3 would fit into that picture. Ultimately the proper solution to improve the upload speeds will be larger pack files, everything else is just too latency sensitive to max out the network connection.

This did not help much.

  • this PR without change → 600M
  • this PR with 128 min-packsize → 800M
  • without this PR → 2G

With the increase in connections and the async PR, we are able to hit an average of 3G with internal and external S3 (only one wasabi location) as well as with the Rest server.

We are unable to get anywhere beyond that. What we have found is that if files look changed (hard link changed, still the same file) and restic does the chunk comparison → this speed is not far off from the average 3G transfer speed.

Are there any options to speed up this chunk comparison or do you have other thoughts on what the new bottleneck is?

→ I am aware that we need to set --ignore-ctime to work around the changing hard-link issue. But it likely runs the same logic for new files and it is possible that this the bottleneck there.

The next bottleneck is probably reading / chunking data. Currently only two threads read and chunk data. Especially the chunking part is limited to a few hundred MB/s per thread. You could try to increase that number in the source code.

restic/archiver.go at b6a38d43b353bb6a30c29d4e233a41471d52924d · restic/restic · GitHub

Besides that there are probably some other bottlenecks somewhere, although I can’t tell where so far :wink: .

Does the backup data set consist of many small files or rather of large files?

That is somewhat unexpected. Although the behavior could change drastically, now that I’ve merged PR #3489 (the async PR). The pack-size PR sort of only increases the pack size and does very little else. At least for a normal HDD as target, larger pack sizes drastically improve throughput as this requires a lot less flushing files to disk. My expectation for rest-server / S3 would have been exactly the same, as both also have to flush files to disk and thus larger pack files put less stress on the IO-subsystem. But it looks like we ran into some other bottleneck somewhere…

mostly large files which are already compressed.

We have large Raid 10 arrays. While HDDs, they are large enough for IOPS not to be an issue.
Also write caching on NVME.

It did help to increase it. The new bottleneck is now at an avg. of 4G