Slow backup with many small files via sftp

Getting Help
1

I currently try to setup a backup with restic to a Hetzner Storage Box via sftp. Unfortunately the backup is very slow. I played around with some settings (-o sftp.connections=10, --pack-size 64) without any success. I made some tests with different subdirectories and figured out, that just a few files work great. Of course of searched for solutions but was not able to find anything.

I use restic 0.16.4 on Ubuntu 22.04.

restic -r sftp::/SOME_DIRECTORY -o sftp.command="ssh SERVER -i /xxx -s sftp" -o sftp.connections=10 backup /SOME_DIRECTORY/ -vvv

open repository
enter password for repository:
repository REPOSITORY opened (version 2, compression level auto)
lock repository
no parent snapshot found, will read all files
load index files
[0:01] 100.00%  43 / 43 index files loaded
start scan on [/SOME_DIRECTORY/]
start backup on [/SOME_DIRECTORY/]
scan finished in 5.748s: 2 files, 2.685 GiB
new       /SOME_DIRECTORY/SOME_FILE, saved in 0.282s (68.720 MiB added)
new       /SOME_DIRECTORY/SOME_FILE, saved in 6.556s (839.764 MiB added)
new       /SOME_DIRECTORY/, saved in 6.560s (0 B added, 0 B stored, 36.310 KiB metadata)
new       /SOME_DIRECTORY/, saved in 6.561s (0 B added, 0 B stored, 297 B metadata)
new       /SOME_DIRECTORY/, saved in 6.561s (0 B added, 0 B stored, 290 B metadata)

Files:           2 new,     0 changed,     0 unmodified
Dirs:            3 new,     0 changed,     0 unmodified
Data Blobs:    467 new
Tree Blobs:      4 new
Added to the repository: 908.553 MiB (65.052 MiB stored)

processed 2 files, 2.685 GiB in 1:32
snapshot 0bcfcece saved

Also a few more small files work still great

restic -r sftp::/SOME_DIRECTORY -o sftp.command="ssh SERVER -i /xxx -s sftp" -o sftp.connections=10 backup /SOME_DIRECTORY/2023/ -vvv

open repository
lock repository
no parent snapshot found, will read all files
load index files
start scan on [/SOME_DIRECTORY/2023/]
start backup on [/SOME_DIRECTORY/2023/]
scan finished in 5.199s: 132 files, 184.043 MiB
new       /SOME_DIRECTORY/SOME_FILE, saved in 0.018s (1.395 MiB added)
new       /SOME_DIRECTORY/SOME_FILE, saved in 0.019s (1.395 MiB added)
new       /SOME_DIRECTORY/SOME_FILE, saved in 0.022s (1.395 MiB added)
...

Files:         132 new,     0 changed,     0 unmodified
Dirs:           23 new,     0 changed,     0 unmodified
Data Blobs:    256 new
Tree Blobs:     23 new
Added to the repository: 178.526 MiB (172.310 MiB stored)

processed 132 files, 184.043 MiB in 1:56
snapshot 7f2c754a saved

But when i choose a directory with many files, it looks like this

restic -r sftp::/SOME_DIRECTORY -o sftp.command="ssh SERVER -i /xxx -s sftp" -o sftp.connections=10 backup /SOME_DIRECTORY/ -vvv

open repository
lock repository
no parent snapshot found, will read all files
load index files
start scan on [/SOME_DIRECTORY/]
start backup on [/SOME_DIRECTORY/]
...
new       /SOME_DIRECTORY/SOME_FILE, saved in 54.361s (729.356 KiB added)
new       /SOME_DIRECTORY/SOME_FILE, saved in 54.360s (730.156 KiB added)
new       /SOME_DIRECTORY/SOME_FILE, saved in 54.358s (730.235 KiB added)
new       /SOME_DIRECTORY/, saved in 62.857s (0 B added, 0 B stored, 3.081 KiB metadata)
new       /SOME_DIRECTORY/SOME_FILE, saved in 54.358s (730.210 KiB added)
new       /SOME_DIRECTORY/SOME_FILE, saved in 54.361s (729.419 KiB added)
new       /SOME_DIRECTORY/SOME_FILE, saved in 54.361s (729.148 KiB added)
new       /SOME_DIRECTORY/SOME_FILE, saved in 0.014s (1.052 MiB added)
...

One minute for <1MB seems to be to much.

Does anybody has advice?
Thanks
Mark

2

I don’t do this myself, but I’ve read that using the REST-server helps a lot. (I have been seriously thinking about it though. Let me know if you try it and it works out.)

3

The storage box I use does not support REST. But it supports rclone. rclone would be my last resort but i would prefer to have solution via the native sftp implementation. I still hope there is just a setting that i need to change.

4

Oh, of course. It’s a storage box. Sorry, I saw Hetzner and thought VPS even though “storage box” is right there.

5

Hey, welcome to the forum.

Unfortunately there is at least one performance bug I’m aware of against the SFTP backend specifically that has yet to be resolved:

Using rclone is suggested as a workaround, as is using an older restic binary.

The above said, it’s probably also worth benchmarking your dataset when backing up to a local repository, just to get a point of comparison. That way you’ll know with more certainty whether the limiting factor is the backend or not.

6

That does indeed sound similar! I found this bug report when I searched for this issue but somehow thought it was fixed. So I think I will just wait for a fix.
Thanks a lot!