I think the unfortunate reality is that this was simply an erroneous estimate, and it would never have been able to complete the backup in 3 hours.
Pulling numbers for the Samsung 990 Pro SSD, reviewers quote around 1400MB/s for sustained write loads (a far cry from the 6900MB/s quoted on the datasheet). That would be ~4hrs to transfer 2.5TB. So even just copying the files, 3hrs would be a stretch.
It’s possible the estimate was so low because you already uploaded the first chunks of data during your “–use-fs-snapshot” run. So perhaps restic was going quickly as it was only encountering data it already had written to the repository. Once it started encountering new data, the estimate changed to be more accurate.
I remember that thread, just skimmed through it to refresh my memory.
I’m probably going to repeat myself here, but 2.5TB is a lot of data.
Thunderbolt saturates in the region of 1000s of MB/s (depending on the version), and the same for NVMe SSDs. However, as we discussed in the last thread, restic isn’t going to get anywhere near that, given it is “chunking” (i.e. splitting and hashing) each changed/new file in the dataset, and during the first backup, all the files are “new”.
I mentioned in the last thread I see about ~230MB/s, using local SSD storage, on a relatively modern (Ryzen 5 5600X) system, and that the speed of restic processing (CPU) is the limiting factor.
You were getting nearer 40MB/s before, and based on the 2.5TB/37hr estimate it looks like you’re now getting around the ~150MB/s mark, which is almost 4x faster. Yay for local storage!
I think I can only really re-iterate what I said last time; the first backup will be slow. Subsequent backups will be several orders of magnitude faster (depending on how much your dataset does or doesn’t change).
If you’re still interested in testing restic, I think unfortunately you’re going to have to get through the first backup before you’ll be able to have any real idea of what the day-to-day performance will be like.
Something that might make the first backup easier, but requires some manual effort, would be to back up the dataset in pieces during the initial backup, instead of all in one go.
Let’s take an example dataset of /mnt/data, where under /mnt/data there are a bunch of subfolders, a through z.
- To avoid having to back up all of
/mnt/data in one go, we can back up the subfolders one, or many, at a time, so that we limit the total amount of data being ingested in each backup run.
- Every new snapshot we take, we add more subfolders, such that by the end of the process, we’re specifying all of the folders under
/mnt/data, i.e. the complete dataset.
- In between each backup, we take a note of the snapshot ID that was previously created, and we tell restic to use it as the “parent” snapshot for each subsequent snapshot. Restic won’t upload the data twice anyway, but this lets the path-based file change detection work effectively.
Our example first backup would be:
restic backup /mnt/data/a /mnt/data/b
And our example second backup would be:
restic backup --parent 12a345b6 /mnt/data/a /mnt/data/b /mnt/data/c /mnt/data/d
Where 12a345b6 is the ID of the previous snapshot we took (the one of /mnt/data/a /mnt/data/b). This can be found by running restic snapshots or read from the previous backup run output.
Repeat this process until all subfolders in /mnt/data have been backed up, then run one final backup:
restic backup --parent a123456b /mnt/data
Where a123456b is again the ID of the previous snapshot.
- After this, you would
restic forget all the “staging” snapshots (those where we specified the subfolders as the backup target).
- From here on, backup the entire
/mnt/data directory, and don’t bother specifying any subdirectories anymore.
- Your next scheduled backup after this should be a good indication of the time this backup will take when run regularly.
The point of this process is to limit the amount of new data ingested, and therefore how long each individual backup takes.
Restic has file change detection functionality, which means the previous data doesn’t have to be fully processed: Backing up — restic 0.18.1 documentation
By splitting the backup into segments, we can leverage that functionality to keep the time taken to complete each backup segment down, while simultaneously increasing how much of the total dataset we have backed up.
Based on you liking the 3hr mark, backing up maybe 250Gb chunks of your total dataset would get close to that (should take ~3:45h), this would require 10 backup runs to do all 2.5TB.
If you can keep the system on for longer, ~500GB should be in the region of a 7.5h runtime, and would require 5 individual backup runs.
Note that all the numbers above are rough estimates based off the 2.5TB/37hr number, which is itself a rough estimate. You may find the system performance varies significantly from this.
unless you were seeing something like a single core pegged at 100% while the others were all idling I suppose 