Possible fix for OneDrive / SharePoint throttling

Recipes
1

Taken from Rclone’s documentation:

Excessive throttling or blocked on SharePoint

If you experience excessive throttling or is being blocked on SharePoint then it may help to set the user agent explicitly with a flag like this: --user-agent "ISV|rclone.org|rclone/v1.55.1"

The specific details can be found in the Microsoft document: Avoid getting throttled or blocked in SharePoint Online

Which means if you pass -o rclone.user-agent="ISV|rclone.org|rclone/v1.55.1" to Restic, it may very well speed up your OneDrive backups!

Please correct me if I put this thread in the wrong place, or if that isn’t the proper way to pass the user-agent. I haven’t tested with Restic yet, but I was trying to download 2.5TB of data for a user and it was going from 60MB/s down to 1-2MB/s after awhile. So far, with the --user-agent string I’m doing a consistent 50-60MB/s. :smiley:

Update: I did eventually get throttled. It definitely stayed at top speed for longer, though. Leaving this here in case it helps someone in the future! I think the real answer is creating your own ClientID for Rclone, but that isn’t something I have access to in my organization.

Second update: It worked great with Rclone alone, but I couldn’t figure out how to get it to work with Restic.

3 Likes
2

according to Preparing a new repository — restic 0.16.3 documentation :

The rclone backend has three additional options:
-o rclone.program
-o rclone.args
-o rclone.timeout

so restic does not support -o rclone.user-agent.
What i ended up doing is add it via -o rclone.args like this:

-o rclone.args=“serve restic --stdio --b2-hard-delete --user-agent ISV|rclone.org|rclone/v1.65.2 --verbose”

The --verbose is for debugging. That runs but stalls on the check snapshots step, same as when I use rlcone without user agent. More trials needed it seems.

FYI - My main motivation is to use rclone in windows and use Files-On-Demand to prevent populating my local drive with OneDrive repo files. Running restic (without rclone) directly on the OneDrive folder works but downloads all files that are used for check --read-data.

3

Thank you for sharing your information, I find this really useful. When you say you have no access to creating a dedicated app ID, what I found is that in some cases, the access is blocked in the GUI, but you can still create your own app-id using powershell.

This can get you started:

You might need to exchange some of the commands to comply with the newer Graph API, I have not looked into that yet.

1 Like