I was hoping to migrate my backup stack consisting of restic + rclone storing backups on B2 to Storj, however Storj imposes a $0.0000088 per file fee where B2 does not.
I have in excess of 300k files and this would work out to $3 extra per month, while storing just shy of 2TB ($8/month at Storj)
My question is, is there a way to force a minimum file size for the files produced by restic such that I could drastically decrease the amount of files stored?
EDIT: slight correction, I have well over 300k files in my backup in B2 currently, not 300k files on my computer.
Yes there is. But not in the release binaries. You either need to make changes to the code, as detailed by @fd0 here:
or make a version with a PR that adds this as an option, such as this one (–min-packsize):
I go the manual-edit route, and make my own executable every couple of releases (128Mb works great for me). If you’re not comfortable with that approach, you probably need to look for a different backup solution (for example, Borg defaults to 500Mb I believe).
The evidence for the need for this in a release version continues to grow (sorry if I’m starting to sound like a brokenrecord )
Thanks a lot, I went the PR route and used 128 as you suggested.
In the future would a restore from the current main branch restic be compatible with a repo created with those very large files?
As I run an initial backup I noticed the ETA is around a fifth what it was with the release version of restic and my upload speed utilization is far better too averaging 810Mbit/s out of 1000Mbit/s provided by my ISP.
I’m also using the RCLONE_TRANSFERS=16 option.
The aforementioned performance improvements might be due to the fact that while I am located in Europe my Storj region is the US (with a backup I care more about solar flares etc than latency) and so restic (or really rclone in this case) doesn’t need to start as many TCP connections when there’s fewer files which might’ve been introducing major latency to the point of hurting the bandwidth.
Good stuff. Yes, particularly on cloud targets or SMR hard drives the speedup is rather impressive.
I recall reading that the change to the repo should be benign. However, that’s not an authoritative answer… But since you’re setting up a new backup, how about trying a restore yourself with a stock release?
If the patched restic version is able to restore the backup then that also works with the release version of restic. In general there shouldn’t be problems except for larger disk/memory requirements in some cases.
There is, however, one corner case mentioned in that PR which hasn’t been addressed: With the larger pack files it becomes possible (although unlikely) to include some many small files in a pack file that the header of the pack file becomes too large for restic to accept it when reading it later on. That said, even in that case it would be possible to just remove/increase the header size limit.
(sorry if I’m starting to sound like a 
)