When backing up to an SFTP repository, I noticed that restic uploads way more data than actually ends up in the repository. For example, a run that claimed to have added 438 MiB to the repository resulted in a transfer of approximately 3 GiB of data to the remote server. I don’t have exact numbers for the storage used on the SFTP server, but it seems to be much closer to the reported 438 MiB.
Is this expected? Is there anything that can be done about this?
The data being backed up is just an nnml directory tree (from Gnus, so lots of small files with messages, and append-only .overview files which can grow quite large over time), and a bunch of mostly unchanging Git clones/source checkouts.
If you have lots of small files, there is quite some overhead within restic. First, a lot of metadata needs to be added (but this should be included in the stats IIRC), but also the crypto and organisational overhead counts for each entry (32 bytes for crypto and some ~100 byte organisational overhead to save SHA256 IDs per unique file content saved).
Can you check repo size before / after adding and check the size, that was actually added to the repo?
I double-checked, and the Added to the repo statistics are just plain misleading. For the latest run, restic said 492.005 MiB there, but actual disk storage increased by 2,681 MiB. That’s not too far off from the number of bytes transferred on the network interface, so it’s probably not network/SFTP overhead that’s causing excessive bandwidth use.
This is with restic 0.9.6 and OpenSSH 8.1p1 on the SFTP server side.
