restic 0.17.1 compiled with go1.23.1 on windows/amd64
I use --pack-size=128 to back up files to OneDrive with Rclone and restic. I back up about 24 times a day, and each time the total size of the files to being backed up is less than 128M, but restic will always create a pack file to store these files. The more backups, the more pack files. I would like to rationalise these pack files so that Rclone has fewer files to check and sync to another remote. I would like to ask if there is a way to use the pack files that are not up to 128M in the repo and fill them up before creating a new pack file?
Consider rustic which gives you much more flexibility in terms of packs sizes.
You can configure following parameters with your repo:
--set-treepack-size <SIZE>
Set default packsize for tree packs. rustic tries to always produce packs greater than this value. Note that for large repos, this value is grown by the grown
factor. Defaults to `4 MiB` if not set
--set-treepack-size-limit <SIZE>
Set upper limit for default packsize for tree packs. Note that packs actually can get a bit larger. If not set, pack sizes can grow up to approximately `4 GiB`
--set-treepack-growfactor <FACTOR>
Set grow factor for tree packs. The default packsize grows by the square root of the total size of all tree packs multiplied with this factor. This means 32 kiB
times this factor per square root of total treesize in GiB. Defaults to `32` (= 1MB per square root of total treesize in GiB) if not set
--set-datapack-size <SIZE>
Set default packsize for data packs. rustic tries to always produce packs greater than this value. Note that for large repos, this value is grown by the grown
factor. Defaults to `32 MiB` if not set
--set-datapack-growfactor <FACTOR>
Set grow factor for data packs. The default packsize grows by the square root of the total size of all data packs multiplied with this factor. This means 32 kiB
times this factor per square root of total datasize in GiB. Defaults to `32` (= 1MB per square root of total datasize in GiB) if not set
--set-datapack-size-limit <SIZE>
Set upper limit for default packsize for tree packs. Note that packs actually can get a bit larger. If not set, pack sizes can grow up to approximately `4 GiB`
--set-min-packsize-tolerate-percent <PERCENT>
Set minimum tolerated packsize in percent of the targeted packsize. Defaults to `30` if not set
--set-max-packsize-tolerate-percent <PERCENT>
Set maximum tolerated packsize in percent of the targeted packsize A value of `0` means packs larger than the targeted packsize are always tolerated. Default if not
set: larger packfiles are always tolerated
Packs smaller than min-packsize-tolerate-percent will be consolidated during prune runs. At least this is my understanding.
Pay attention that rustic is still in beta. From my experience (as I’ve been using it for many months now) data is safe but some features do not work yet 100% as epxected.
rustic author here.
Acutally I think this topic is nothing where rustic can make it better than restic, as @yufoq already decided for a fixed target packsize, which is both possible with rustic and restic.
@yufoq About your question: No, there is no way to “fill up” non-full pack files in a future run. But you want to read the docu for the prune command which is able to repack small pack files (restic prune --repack-small).