Backing up from SFTP is slow for 200k files

Getting Help
1

Hello!

I’m currently setting restic on a new machine in order to backup multiple servers around. I’m using rclone to connect to these servers using FTP/SFTP and mount it in a local folder, then use restic to backup that folder (I use a single repository, and use tags to keep track of sources).

restic 0.17.0 compiled with go1.22.5 on linux/amd64

I’m backing up TO Dropbox (also using rclone, but that’s not the issue here).

I have no problem backing up servers with 50k files. However, one of my servers has more than 200k files (which total around 20G), and the first backup took 5 full hours.
I expected the next backup to be way faster, but it still took 2 hours, even though not many files changed, here’s the JSON summary:

[{"message_type":"summary","files_new":501,"files_changed":676,"files_unmodified":200251,"dirs_new":632,"dirs_changed":6584,"dirs_unmodified":46175,"data_blobs":1164,"tree_blobs":2647,"data_added":199813762,"data_added_packed":35367955,"total_files_processed":201428,"total_bytes_processed":18154372320,"total_duration":5051.145794897,"snapshot_id":"c85bc495abf1aad19c2fec7698c1001fc7e52ade7519d09731726aba9bfffc8a"}]

I’ve read on some other topics that it could be because FTP/SFTP is quite slow, even if it’s just for listing updated/new files, is that true? Would there be a very significant difference if I was running restic directly on that server? (note that I currently don’t because 1. I would need to install rclone+restic on all servers 2. I do not want to impact CPU of other servers too much, eg with compression)

Here’s the command I’m currently running:

restic -r rclone:Dropbox:/ResticBackups backup /home/ubuntu/backups/tmp --tag $repo_name --group-by tags -o rclone.connections=10 --read-concurrency 10 -p /home/ubuntu/backups/password --json --no-scan --pack-size 64 --skip-if-unchanged | jq 'select(.message_type == \"error\" or .message_type == \"summary\")' -c

Could anyone help with that, and possibly suggest solutions if you’re aware of any?

Thanks,
Gurvan

2

Restic has to check the file metadata of every file. The rclone setup likely causes rclone to retrieve the file metadata sequentially for every file. As every roundtrip to the server takes a few milliseconds or more, this quickly sums up to several hours.

When restic runs locally on a host, it usually takes only a few minutes to scan one million files.

3

Then that is indeed the very bottleneck I’m encountering. Unfortunately I haven’t found yet a solution to that, since I want a centralized way of making the backup. Maybe SSH-ing into the VPS, transfer the restic binary + rclone binary + rclone config (backing up to dropbox) could reduce the time needed?
Though, that would mean the backup is done on each server, and therefore it uses these servers CPUs…

4

When connecting via SFTP, then the server-side component of SFTP also uses CPU on those servers. Although it’s probably less than what running restic natively would use.

Whether that is feasible also depends on whether you can trust the VPS with the dropbox credentials.