I’m having a degraded performance while restoring a backup from S3 bucket (ceph+radosgw). The restore seems to start well but at some point the performance is heavily degraded making the restore to last hours. You can see the pattern on the attached plot.
I have a feeling that could be related when restoring big files. As an example, restoring a 170G repository takes a few minutes to restore 30G and several hours to restore the rest which I think that this time belongs to two 70G files that are in the folder (because I did a manual check on the restore folder). On the other hand, restoring a backup of 50G and 1.3M files took 23 minutes with no apparent slow down.
this is the command I use to do the restore (last snapshot). I don’t know if could be related, but for the backup I used the
restic --cache-dir /var/cache/restic/ -o s3.connections=64 restore cf6ecdfb --target /var/tmp/restic/
Changing S3 connections helps in the initial phase, but the degradation still happens. I’ve tried several times so I don’t think that is related to the load of our S3 endpoint.
Any idea or someone had this same issue?
Thanks a lot !!!