Degraded restore performance (S3 backend)


#1

Hi all,

I’m having a degraded performance while restoring a backup from S3 bucket (ceph+radosgw). The restore seems to start well but at some point the performance is heavily degraded making the restore to last hours. You can see the pattern on the attached plot.

I have a feeling that could be related when restoring big files. As an example, restoring a 170G repository takes a few minutes to restore 30G and several hours to restore the rest which I think that this time belongs to two 70G files that are in the folder (because I did a manual check on the restore folder). On the other hand, restoring a backup of 50G and 1.3M files took 23 minutes with no apparent slow down.

this is the command I use to do the restore (last snapshot). I don’t know if could be related, but for the backup I used the --parent flag.

restic --cache-dir /var/cache/restic/ -o s3.connections=64 restore cf6ecdfb --target /var/tmp/restic/

Changing S3 connections helps in the initial phase, but the degradation still happens. I’ve tried several times so I don’t think that is related to the load of our S3 endpoint.

Any idea or someone had this same issue?

Thanks a lot !!! :slight_smile:


#2

There’s been some discussion about this topic on Github #2074, but AFAICT no solution yet.


#3

Thanks!, I will follow-up this on that issue.


#4

Alternatively u can try with rclone
Config the S3 backend on rclone and use restricted with rclone:S3 backend