restic 0.19.1 compiled with go1.26.4-X:nodwarf5 on linux/amd64
I made a new repo. Say that I did not use “restic init --copy-chunker-params”, but just made a new repo… THEN, I backup a snapshot from a already backed up snapshot using (–from-repo). I realise that I may not get full deduplication… but is there any way in the future to gain back deduplication (in future backup/snapshots) when I for example go into older snapshots and delete files I don’t want anymore, and then use commands such as “prune”, and “forget” (if in fact I decide I no longer want those previously backed-up files?), and then the newer made repo will also copy the newer snapshot and realise that there are (were(past tense)) deleted files, and then I can run “prune” and “forget” on the newer made repo (the newer one without the “restic init --copy-chunker-params”), and it will just prune the older files anyway, so kinda it does not matter if I initiate the new repo with “restic init --copy-chunker-params”, but I can still deduplicate everything eventually by pruning and forgetting old files which I don’t want?.. Thanks everyone, if you understand my question… I can re-iterate or explain better if needed. Thanks!
long question, but short question is "I can always make a repository deduplicated as long as I use “prune”, and “forget”, right?.. even if I initially did not use the “–copy-chunker-params” on repo initialisation? (and used the copy --from-repo) command? Thanks if that was easier to understand…
…a further breakdown… I made a new repo… not with “–copy-chunker-params”. I use the “copy --from-repo” command to the new repo. I know it will not be deduplicated to the fullest, but if in the future I do a straight “backup” command… will it deduplicate?
Have a look at PR Rechunk copy feature by dgkmd · Pull Request #5529 · restic/restic · GitHub. This is an excellent method of copying between repos which have no shared --copy--chunker-parms. I have used it for many months now. It shows excellent performance characteristics and does a perfect job of copying snapshots and using deduplication at the same time. However, there might be a penalty involved, where you are paying €,£,$ for your read/write access to your repositories, since copy --rechunk reads all the selected files completely from source and re-encodes them with the target chunker-parms. In the long run having repositories with the same chunker-parms is the better option.
Consider the amount of work you have to spent once to create a repository with identical chunker-parms vs. the loss of deduplication and therefore a larger size repository forever. The choice is yours!
Does your version of restic wplapper/restic have the added --rechunk option? I’m not sure how to patch code.
Please don’t patch the restic code, start with a fresh copy.
My repository does not contain the code. But the best thing you can do is to go GitHub - dgkmd/restic: Fast, secure, efficient backup program · GitHub, clone the code into a new directory on your local machine and run make. I am assuming you have the latest go version installed. Otherwise install go from https://go.dev/dl/go1.26.6.linux-amd64.tar.gz.
dgkmd is the author of this PR which contains the “copy --rechunk” code.
in detail:
cd where ever you want to be, e.g home directory, then
mkdir dgkmd
cd dgkmd
git clone GitHub - dgkmd/restic: Fast, secure, efficient backup program · GitHub
cd restic
git fetch origin
git switch feature/rechunker
git status //make sure that you are in branch “feature/rechunker”
make
export RESTIC_FEATURES=rechunk-copy=true
./restic -r target-repository copy --from-repo source-repositoty --rechunk -vv
your new restic module is in dgkmd/restic/restic
Look at comments in changelog/unreleased/issue-5473. It explains in more detail how copy --rechunk works and how to use it.
Well, thank you. I’ll give it a try. Thanks for the detailed instructions also. I think this would be a nice thing to add in the official repo.
Hi Marinaro, every PR (=pull request) should eventually end up in the official restic repository. It may however take some time before this will happen. It very much depends on the length of the review queue for all these PRs and the time it needs to review each of these PRs.
Worked great, thanks. This (feature) is what I’d been looking for. It compiled, and I’ve used it successfully. Thanks for the detailed instructions also. Been reading the conversation between you and dgkmd on the github. Great job. Thanks again!