Backup Strategy - Help

[returntrip]

I am currently using Crashplan and looking at moving to restic. I basically need some help figuring out the best way forward to achieve the following:

a) Have one backup copy which is pruned (by pruning I mean, all files deleted are removed from the backup on a schedule)
b) Have one backup copy which is NOT pruned and contains all historical data (just in case!)
c) Have one onsite storage with (a)
d) Have one offsite storage with (b) and possibly (a)

My idea is to use the following servers scheme but I am not really sure what is the most efficient way to achieve my requirements above. I do not know if I should:

• create both restic pruned and unpruned backus on the primary server and then copy them with rclone (to the respective servers)
• or maybe create the restic backup, rclone it to the offsite server in append only mode and to the on site server without using append only mode? What would happen to both rclone backups when I prune the restic backups?

1. Primary on site backup: using restic, backup my server content to a separate HDD attached to the server
2. Secondary on site backup: using rclone, copy the backup from point (1) to a secondary server
3. Tertiary off site backup: rclone, copy the backup from point (1) to an offsite server

As you can see I am quite confused and would appreciate your expertise to crete a reliable backup strategy.

Thanks in advance!

[akrabu]

You may find this thread interesting, especially @cdhowie’s setup:

Backing up round robin to initially cloned repositories <-> local restic cache

I’m not sure if the cache is totally rebuilt as in downloading all index files (@fd0 may be able to answer that) but it would have to at least download new index files and remove local copies of missing ones or very basic functionality would break in any situation where multiple restic clients share access to the same repository. My point is just that what is being described here would look exactly the same to restic as though the same repository was just modified by another client. Either bot…

[returntrip]

Thanks I will have a look!

[returntrip]

I have less than 600GB of data to backup and its data changes very little so I guess I was overthinking my strategy.

To replicate Crashplan (except for the missing always on backup), I guess it will be sufficient to have a single restic backup (with no forget/prune) and then rclone the restic backup to both the onsite and offsite servers.

Of course, feedback and best practises always welcome!

[tomwaldnz]

Suggest you use the Grandfather / Father / Son backup rotation scheme.

In short, it will give you something like this:

• Daily backups, kept for the time you specify (a week or a month)
• Weekly backups, kept for let’s say eight weeks
• Monthly backups, kept for say 24 months
• The current version is obviously never deleted

That way if you notice a problem within a week you can roll back to any daily version. If you don’t notice for six weeks you can roll back to a weekly version. And so on.

I use a command something like this for my purge

restic.exe  --repo s3:bucketname forget --keep-daily 7 --keep-weekly 8 --keep-monthly 24

[returntrip]

Thanks, I think that could work for me, I will take a look at that after figuring out how to speed up rclone checks.

I have currently setup restic to run every 2h (without forget atm) and backup my data to a local drive, after restic finishes the backup, rclone syncs the backup to a remote server, everything seems working fine except for rclone taking ages running checks before syncing (over 1h checks for 600GB). I have created an rclone cache but that does not seem to help much, but I guess I will have to ask for help on the rclone forum

[returntrip]

@tomwaldnz Any special consideration re forget policy If I am running backups every two hours?

Would something like this be OK (want to keep a few extra snapshots for the day):

forget --keep-last 5 --keep-daily 7 --keep-weekly 8 --keep-monthly 24

Also, one thing I am not understanding is this: assuming I do not want any data purged from my backup, is there away to “compact/merge” all snapshots in a single snapshot which contains all historic data (from the past snapshots until the most current version)?

Apologies if it is a stupid question.

[tomwaldnz]

I’m a user not an expert on restic, so this is just from what I know. The command line looks ok to me, but I haven’t used “keep last” myself.

If you don’t want any data purged from your backup don’t use the “forget” command. Restic is block based, snapshots just point at blocks. As far as I know you can’t compact anything, you just forget the blocks you don’t need which are purged.

[cdhowie]

@returntrip

Also, one thing I am not understanding is this: assuming I do not want any data purged from my backup, is there away to “compact/merge” all snapshots in a single snapshot which contains all historic data (from the past snapshots until the most current version)?

@tomwaldnz is correct: just don’t forget any snapshots. There isn’t some more efficient format that restic can use to “compact” a set of snapshots; a snapshot is just a label that points at a “tree” (directory) object in the repository, which is the root directory of the snapshot. Tree and data blobs are already deduplicated.

Quite simply, there isn’t anything to compact.

[returntrip]

Thanks for the further confirmation. I guess I was thinking about the hundreds of shanpshots I will have (cause I am doing backups every two hours) so I am dreading at how long the output of the snapshots argument will be in a few weeks/months .

I guess, after all, I might be forgetting/pruning snapshots for my own sanity.