Degraded restore performance (S3 backend)

Hi all,

I’m having a degraded performance while restoring a backup from S3 bucket (ceph+radosgw). The restore seems to start well but at some point the performance is heavily degraded making the restore to last hours. You can see the pattern on the attached plot.

I have a feeling that could be related when restoring big files. As an example, restoring a 170G repository takes a few minutes to restore 30G and several hours to restore the rest which I think that this time belongs to two 70G files that are in the folder (because I did a manual check on the restore folder). On the other hand, restoring a backup of 50G and 1.3M files took 23 minutes with no apparent slow down.

this is the command I use to do the restore (last snapshot). I don’t know if could be related, but for the backup I used the --parent flag.

restic --cache-dir /var/cache/restic/ -o s3.connections=64 restore cf6ecdfb --target /var/tmp/restic/

Changing S3 connections helps in the initial phase, but the degradation still happens. I’ve tried several times so I don’t think that is related to the load of our S3 endpoint.

Any idea or someone had this same issue?

Thanks a lot !!! :slight_smile:

There’s been some discussion about this topic on Github #2074, but AFAICT no solution yet.

Thanks!, I will follow-up this on that issue.

Alternatively u can try with rclone
Config the S3 backend on rclone and use restricted with rclone:S3 backend

Just moments ago https://github.com/restic/restic/pull/2195 was merged into master. Please try if you are able to – you can download a build of the latest master, which at the time of writing includes the merge of PR #2195, from https://beta.restic.net/?sort=time&order=desc . I’m mentioning this because the changelog for that PR mentions this forum thread :slight_smile:

1 Like

Thanks a lot guys for all the work to make this merged !
I’ll give it a try!