Hi,
I recently compiled my own binary in order to add a couple of features that aren’t in 0.9.6. I’m very happy with the resulting performance-improvement, but have no sense for whether such an approach could risk the soundness of my backups (there’s no point improving performance if you compromise your ability to restore!).
There’s probably no black-or-white answer, but I’d appreciate any opinions or advice from the many experienced people here: how safe in general do you feel it is for a relatively-competent-non-expert to be venturing beyond the release binaries? Is this something that you’d simply never advise with a restic-focussed backup solution? Or, in combination with things like restic check --read-data
, do you think there is minimal increased risk?
Details of the changes and benchmarks follow if you’re interested, although this isn’t the main point of the question.
Starting from 0.9.6, I added:
- Larger pack sizes (following @fd0’s instructions here). I understand the historic reasons for a 4Mb default, but suspected this was a bottleneck in my use-case, particularly with SMR drives
- Copy functionality as per PR#2606.
For those, like me, with no previous git or go experience, here’s how I did it:
details
Get the master branch:
git clone https://github.com/restic/restic
From within the generated directory, get the desired pull request:
cd restic
git fetch https://github.com/restic/restic refs/pull/2606/head:pr2606
git checkout pr2606
Change the minPackSize, by altering line 39 in internal/repository/packer_manager.go
to (in my case):
const minPackSize = 128 * 1024 * 1024
Then compile for my current system:
go run build.go
and for windows:
go run build.go --goos windows --goarch amd64
And here’s the impact on my NAS-based repo. The desired SMR-copying speedup is better than hoped: times are now comparable to the PMR reference drive. I suspect big prunes may turn out to be slower, but I’m happy with that trade-off.
operation | 4Mb minPack | 128Mb minPack |
---|---|---|
backup (minimal changes) | 35 min | 35 min (0%) |
empty prune (no changes) | 110 min | 27 min (-75%) |
check 1% of repo | 90 min | 86 min (-4%) |
rsync-to-USB (Seagate 1Tb SMR, from blank) | 54.2 hrs | 10.3 hrs (-81%) |
rsync-to-USB (Seagate 1Tb SMR, no changes) | 88 sec | 7 sec (-92%) |
rsync-to-USB (WD 8Tb, from blank) | 12 hrs | 10.4 hrs (-13%) |
rsync-to-USB (WD 8Tb, no changes) | 84 sec | 4 sec (-95%) |
directory | size before | size after | files before | files after |
---|---|---|---|---|
data | 816G | 823G* | 171,318 | 7,124 |
index | 212M | 204M | 58 | 341 |
keys | 4.0K | 4.0K | 1 | 1 |
locks | 4.0K | 0 | 2 | 0 |
snapshots | 1.4M | 1.4M | 341 | 341 |
total | 817G | 824G* | 171,721 | 7,808 |
*The first prune on the newly-created repo (a restic copy
of the original) found 31,220 duplicate blobs. That explains the increased repo size after copying (the sizes matched, once the prune was complete). I can’t think why the duplicates should exist, so wonder if this is either a bug in PR#2606, or a problem caused by the change in minPackSize?