Using Ubuntu 21.04. Just finished my first successful Restic backup of my 1TB (210k files) user files from my laptop to a USB connected spinning hard drive.
I’m not a techie but I get the impression that Restic can safely handle more incremental backups than rsync. So how many incremental backups can I reasonably do with Restic? Assume about 400 files will change between each incremental backup. (210,000 files total. A mix of small and large files.)
My initial backup took 43hrs. So I’m hoping that Restic can safely do a high number of incremental backups before I need to start all over again with another full backup. Let me know.Thanks!
Both rsync and restic can handle as many backups as you have drive space. However, rsync just makes your backup exactly equal to the source so you only ever have the last stand to go back to. On the other hand restic will keep a history so that you can go back to what the source looked like at the time of each backup run even if you have backed up many times inbetween. When you mount a snapshot, you will see the file system at the point of the backup.
This means that your repository will steadily grow as old and new files are kept in it. To counter the drive filling effect of this, you can regularily have restic forget certain snapshots according to rules you give it. Check this out if you want to learn more:
Great. Thanks. Soon as I get a moment, I’ll read the docs.
It sounds like I can indefinitely keep running Restic incremental backups. And when the drive gets full just delete the oldest snapshots.
So am I correct in thinking that (for this particular repo) I won’t ever need to do another full backup as long as I keep deleting oldest snapshots? And making sure there’s always space on the drive?
So even if I’ve done a hundred or more incremental backups, Restic can safely and efficiently keep track of everything without getting overloaded?
I like the ability to go back in time to get old versions of files if needed. Nice.
The concepts of full and incremental aren’t a great fit to Restic’s model. Every backup is a “full” one, but if not much has changed, it should be a LOT quicker than the first one. To answer with an example of the same order-of-magnitude as yours:
My main repo is currently 1.4tb in size. It contains 771 snapshots from daily backups over the last 2+ years. The directories being backed-up are currently of the order of 2.8Tb (the joy of deduping!), containing 587,000 files. This is running on an rpi4 4gb over a network. Each overnight backup currently takes about 16mins.
More snapshots lead to more disk usage, restic consuming more ram, and repo operations taking longer. I was therefore expecting to need to have a snapshot-retention policy, but so far everything is nicely within my system’s limits. It got hairy for a while with an rpi3 and 1gb ram, and the number of tiny repo files were a killer when rsyncing my repo to USB until I rolled-my-own executable with larger pack files. Other than that, it’s been hassle-free.
So with your system, you should be fine. A higher priority is probably to setup occasional checks of your repo with the --read-data option (you don’t need to read the whole repo each time, just loop through subsets), and get a second copy of your repo on something else.
@m3110w Given this and previous threads I think you would benefit from forgetting about your previous thoughts of how backups work, start with a clean mental state and no assumptions, and then read the restic documentation at restic.readthedocs.io, to better understand how simple restic backups really are.
Good point. Will be good to learn a bit more how Restic differs from something like rsync. Now just gotta finish some projects to find some reading time. Too many things to do!
My repo have 24TB in size. I need make 2 snapshots for daily for 3 years or more. I’m afraid of having to make a rebuild-index in all this data structure. What’s the best strategy for this?
The runtime of rebuild-index only depends on the size of the repository, not the amount of snapshots. I’m not aware of any reason to run rebuild-index with recent restic version except for data corruption/loss at the storage backend.
To have 2190 snapshots or more with 24TB is safe? Is the probability of corrupting all data low or high? Corrupt not by hardware and file system failure, but by power outages or internet link down.
