Degraded restore performance (S3 backend)

Getting Help
1

Hi all,

I’m having a degraded performance while restoring a backup from S3 bucket (ceph+radosgw). The restore seems to start well but at some point the performance is heavily degraded making the restore to last hours. You can see the pattern on the attached plot.

restic_restore
restic_restore.png1189×240 25.1 KB

I have a feeling that could be related when restoring big files. As an example, restoring a 170G repository takes a few minutes to restore 30G and several hours to restore the rest which I think that this time belongs to two 70G files that are in the folder (because I did a manual check on the restore folder). On the other hand, restoring a backup of 50G and 1.3M files took 23 minutes with no apparent slow down.

this is the command I use to do the restore (last snapshot). I don’t know if could be related, but for the backup I used the --parent flag.

restic --cache-dir /var/cache/restic/ -o s3.connections=64 restore cf6ecdfb --target /var/tmp/restic/

Changing S3 connections helps in the initial phase, but the degradation still happens. I’ve tried several times so I don’t think that is related to the load of our S3 endpoint.

Any idea or someone had this same issue?

Thanks a lot !!! :slight_smile:

2

There’s been some discussion about this topic on Github #2074, but AFAICT no solution yet.

3

Thanks!, I will follow-up this on that issue.

4

Alternatively u can try with rclone
Config the S3 backend on rclone and use restricted with rclone:S3 backend

5

Just moments ago https://github.com/restic/restic/pull/2195 was merged into master. Please try if you are able to – you can download a build of the latest master, which at the time of writing includes the merge of PR #2195, from https://beta.restic.net/?sort=time&order=desc . I’m mentioning this because the changelog for that PR mentions this forum thread :slight_smile:

1 Like
6

Thanks a lot guys for all the work to make this merged !
I’ll give it a try!