Mirror Snapshots from One Backend to Another

Features and Ideas
1

Posit the following scenario: I have a restic repository locally on a NAS, but I want it mirrored to B2.

I think ideally what I’d want is something that looked like…

restic mirror --from-repo local --to-repo b2 $snapshot

And this would compare the blobs on both sides and simply move the blobs necessary to make $snapshot complete on the target host. After the command was finished, the snapshot in local would appear in b2.

I realize this is possible with mount or some creative scripting but this way seems the most airtight and least prone to error, also most other solutions I can think of would require rehashing the files.

…oh wait, this doesn’t work because each repo has its own polynomial for the Rabin fingerprints, right?

2

This is precisely what rclone is for: https://rclone.org/ :slight_smile:

Except it works with any files, not just restic repos (but it works very well for restic repos, and I use it for exactly the purpose you describe).

2 Likes
3

Have you ever tried mc, minio client?
It does what you need.

Here you can find the doc:
https://docs.minio.io/docs/minio-client-complete-guide#mirror

We use it for replicate/mirror s3 repo from one server to another

Bye

4

Would either of these solutions be able to copy specific snapshots from one repository to another? It just seems like it will just copy the whole thing. I suppose that’s mostly okay.

5

So you want a snapshot “mirrored” to another repository, not the repository mirrored to another location.

In other words, you want to merge (partial) repositories.

I don’t believe restic supports this natively… I wonder if you could hack it together by copying the files into the other repository and then run rebuild-index and prune?

6

Right, but just copying the files won’t work, because repos initialize their content defined chunking differently, no two repos will ever agree on the Rabin fingerprint for a given bytstream. The data has to go through and entire restore process and then backup again, so maybe this ask isn’t practical.

7

Hmmm, yeah, and even then the encryption keys would probably be different, unless you went to efforts to make them the same.

8

Any reason not to simply backup to the 2nd repository?

9

Well, suppose the files that live in the snapshot I’m trying to export aren’t on my drive anymore, like I’m trying to move historical snapshots that live only on the backup.

Or I’m trying to do the export with an agent that doesn’t have access to the original files, like a cloud server process somewhere that can’t just ssh into my office machine and kick off restic. Or, relatedly, my office system is switched off or unavail.

10

You are right. Currently these use cases aren’t covered very well. One approach would be, as you mentioned, to mount and go from there. But that isn’t very convenient if you have lots of snapshots.

11

The thing with mount into backup is that the files have to go through the slow rescan process all over again, but again if you’re moving between two distinct repositories I guess this is unavoidable.