Backing up to repo on OneDrive via rclone is very slow

Getting Help
1

Hi,

I have a restic repo with roughly 900 GB of data that I recently moved from a SFTP storage to OneDrive. While the restic repo was accessed directly via SFTP, taking an incremental backup of 300 GB of data took ~15 minutes (way less than 1 GB of the data actually changed). When accessing the repo on OneDrive via rclone, taking an incremental backup of the same 300 GB data now takes around 3 hours.

I use this command for taking the backup (restic version is 0.16.0):
restic -r rclone:onedrive-backup:restic --password-file /config/password.txt --host foo backup /mnt --verbose --verbose --ignore-inode --no-cache --no-scan

Looking into the verbose log I found that for OneDrive it seems to take restic way longer to find out wether a file has changed: For an example folder in the 300 GB data set, it took 5 seconds with SFTP, while it now takes 100 seconds with OneDrive via rclone to find out that none of the files in the folder have changed.

Is there any way to speed up backing up to OneDrive?

2

What you are seeing is probably the result of using --no-cache. To check a folder for changes, restic first has to download the tree blob from the last snapshot, which means one backend access for each (!) folder in the backup. If the latency to retrieve a tree blob increases from a few milliseconds to a few hundred milliseconds then the result is a massive slowdown like what you are seeing.

The cache is absolutely essential for good performance, if the latency to retrieve files from the backend is higher than a few milliseconds.

1 Like
3

Thanks, removing --no-cache did the trick. I totally forgot that I added that flag to save some disk space. Removing it turned 3 hours into 3 minutes for backing up the (unchanged) 300 GB data set.

2 Likes