Efficient backup in serveral tranches

1

Hi restic fans! I love restic, but I have a problem that I don’t know if there’s a better solution than what I’m doing.

I make a backup over an asymmetric link, which has such a small upload bandwidth that the initial backup takes two weeks according to ETA. And unfortunately it is not possible to run this continuously, because the backup cannot run at night.
Plus the hard disks from which the backup is made are not very fast either.

It’s clear that restic handles interrupted backups well, but after a day of backup of course a complete snapshot is not taken and therefore if I start it again the next day, it spends many hours with re-reading and checksumming the data it has already uploaded the day before, and only then starts uploading the new data. This is a very inefficient solution, because moving forward in time, more and more time is spent exclusively with rechecking the previous data, leaving only a small amount of time for uploading new data.

Therefore, what I currently do is try to break the material to be backed up into smaller pieces, as large as can be uploaded in a day, to make a complete snapshot of them, and then refer to it as -parent shapshot, so that it just runs a stat() over it on the next day without rereading the content.

For example, let the tree to be saved be
/tobackup/a01/…
/tobackup/a02/…
…
/tobackup/z99/…

Unfortunately, it’s not a good solution if I just save the a01, a02 directories one by one each to a snapshot, because when everything is uploaded and I do the complete

restic backup /tobackup

then I can’t specify all the previous snapshots as the parent snapshot. And because of that it wants to check everything again, which doesn’t fit in a day (only the reading is longer than a day).

So what I do now is to incrementally save them:
restic backup /tobackup/a01
restic backup /tobackup/a0[12] --parent prev_id
restic backup /tobackup/a0[123] --parent prev_id
…
restic backup /tobackup/a0[1-9] --parent prev_id
restic backup /tobackup/a0[1-9] /tobackup/a11–parent prev_id
restic backup /tobackup/a0[1-9] /tobackup/a1[12] --parent prev_id
…
restic backup /tobackup --parent prev_id

This way no checksumming of earlier files occurs, but it’s a pain in the ass carry out the whole procedure, especially because I have to check the size of the next backup each time and the directory structure is not so nice as in the example.

Features that could help:

  • merging already existing snapshots. With that I could simply merge all previous snapshot instead of doing a full backup. But I found the thread, the idea was rejected.
  • smart recognition of many parent snapshots. AFAIK it is not yet implemented.

Is there any easier way to perform a long backup with many interruption than what I’m doing now?

2

You could use rustic (disclaimer: I’m the author) for merging snapshots.

3

thx! I’ll give a try.

4

You did not say how big the data is. One thing I do is backup to a local restic repo, and let syncthing take care of copying that to another laptop which is not up all the time and yet I need it to be one of my backups.

Caution: the very first backup needs to complete before you can expect the destination to be useful for recovery. But after that, at least in my case, the incrementals get uploaded well before the next backup rolls around. I do check the destination by locally running a restic check --read-data

5

Yes, that would indeed be a solution, but it’s remote, I don’t have physical access, I can’t bring extra media. But backing up to a local repo (remotely) and then gradually synchronizing it might really be a viable way, thanks for the idea.