Increasing transfers with Restic (rclone end point)

Getting Help
1

Is there a way to increase the number of simultaneous transfers within Restic whilst using a Rclone end point? I see some references in the search to this but no syntax.

I separately use Rclone for other tasks and use the --transfers x flag, but I have tried various forms of this within the transfer to no avail.

This is the syntax I use (without any attempted rclone modification) for Dropbox. I am happy to receive other tips to modify this backup job if they exist. I’m relatively new to Restic.

restic -r rclone:DR:restic/Appdata --verbose backup /Volumes/Appdata

Thanks !

2

I think you may be looking for the -o rclone.args option. It is described in the section on rclone in the docs.

Having said that, I’m not sure how exactly restic uses rclone and whether the decision to perform multiple simultaneous transfers or not would lie with restic or rclone in this case :see_no_evil:

3

Thanks. I think I had seen that page before when browsing to set up things, but maybe the detail got lost in the mass of information.

The example helped but it didn’t show the backup command. In the early morning without much coffee that caused a bit of head-scratching until I started writing this response and detailing what I did and what did not work :slight_smile:

This is the command I have now run, but it does not show at least more than two files on the terminal screen at a time, so I don’t know if it is really transferring more than two.

restic -o rclone.program="/usr/local/bin/rclone" \
  -o rclone.args="serve restic --stdio --transfers 10 --verbose" \
  -r rclone:DR:restic/GTW backup /Volumes/Videoserver/GTW

With the size of the backup jobs it would be very desirable to get things going with a little boost!

Thanks.
d

4

I see yes… I tried to look for some info on this but couldn’t really find something. I guess it is restic which decides the limit of how many files are processed at a time, but I will need to defer to the experts on this one :slight_smile: sorry…

5

Thanks. I appreciate the nudge in the right direction.
Do the developers/experts frequent this forum often, or is this something perhaps to ask on Github?

6

I do believe they frequent the forum yes, I have received some very helpful replies from some of them. Let’s see whether one of them can provide some answers…

7

See restic options and try applying the rclone.connections option (default value is indeed 5), does it seem to help?

8

Thanks. I will do some deeper searching/experiments over the weekend. Google didn’t immediately cough up a man page or similar for the given reference.

9

If you’re backing up lots of small files then the number of transfers is currently limited to two. There’s a PR (Asynchronously upload pack files by MichaelEischer · Pull Request #3489 · restic/restic · GitHub) to fix this, but it will probably take some time until can be merged. For large files there should at most as many concurrent uploads as you have CPU cores. Increasing the amount of transfers/connections beyond that won’t make any difference.

10

Thank you for the reply. Sorry for the delayed response. The mail program hid the forum’s flagging alert in a different tab than “main” this time. Intelligence, huh.

I’ll be honest and I wouldn’t have a clue what to do to make the changed file appear within my installation. I am still learning the power of Restic (and the excellent Resticprofile script/app). If it is simple (to you) and easily documentable, I’d be happy to give it a try. Would I then need additional flags, or do the changes appear in the default mode.

I back up to Google Drive and Dropbox via rclone’s mount.

D

11

You could try adding a -o rclone.connections=10 parameter to the restic call. However, I don’t know how to pass that parameter on to restic when using resticprofile.

Please make sure that you’re actually able to restore your backups. There have been quite a few problems using these backends, just search for them on the forum. My guess would be that the problems start to show up for larger repositories ranging in the hundreds of GBs, but maybe also earlier.

12

Thanks. I can give it a try.

In Resticprofile rclone is passed by the following configuration file (extract):

repository = "rclone:DR:restic/Darren"
option = [

    "rclone.program=/usr/local/bin/rclone",

    "rclone.args=serve restic --stdio --log-file=/Users/di/.restic/Darren-log.txt --log-level INFO"

]
I do intend to investigate the issue of restore once some repositories have uploaded. Especially as some will be in the Tb range and higher (archiving media files).

13

Is there something new about this? I’m serving rclone by myself with --transfer set and then running restic with rest:ip:port but the transfers are always 4 or 5 and I get slow speeds, it doesn’t go above 10 MB/s when it should reach 70