Hi all. After some years with Restic I have some feedbacks (as a regular user).
First… some background. I have close to 3 Tb of data. I run restic over rclone to an SFTP server on which I sync to cloud automatically (this cloud is client based encrypted).
This is tedious, since each Restic run takes minutes, sometimes hours for each update / snaptshot. I have SSDs and it’s optimized… well… I think this is what it is. No complains.
I never had to restore anything from cloud. Since it’s possible (I tested my process)… it’s complex and I was not sure about the way of just recover part of the backup… not all.
It’s expensive in terms of space… dedup is not helping me here.
So… I’m thinking on moving to less but faster backup & encryption system based on symmetric key encryption on Rclone profile:
It should be faster.
I’ll only sync modifications… no need to read all 3Tb for a small diff.
I more or less understand where the file that I want to backup is… and actually I can perform ls etc without de-encrypt the whole thing.
Downside > Encryption is more weak.
Take in consideration that I have another encryption layer and that I can use quite long symm encryption keys… I think in overall is positive.
So…
First… thanks for read up to here
Second > is this really a good idea? Any feedback?
My setup is different, I don’t use rclone but backup to locally mounted drives and to rest-server backends. I separate the large backups (pictures, videos) from the frequent ones (home). My total backup size is also about 3 TB. With that setup, my experience is quite different:
Processing ~200 GB of uncanged data takes only a few seconds.
Processing hundreds of thousands of files takes only seconds if only a few are changed
When new files are added (like 100 GB new videos), it takes about as long as it would take to copy them over (plus some reasonable overhead).
Neither deduplication nor compression help in terms of size and speed with large video files as they are already effectively compressed. But that’s what I would expect.
So I suspect it’s more the 2-step approach with restic and then rclone which makes things slow for you?
Why Rclone should add latency?. I’m fine eliminating it. I don’t recall why I start using it… but I can go over SFTP. I think I tried to avoid putting user&pass on a local file… anyway… I can change.
Should be Rclone responsible of this very long time on each backup?
Also… I’m still worried on how to extract partial backups from Restic… Files are “spread” and I don’t know which of them I need in order to have a partial backup…
Are you ok in case you need to recover one or multiple files from backup? Which is your process?
I don’t think so, but you can try out if this helps.
I would actually think that your setup has some problems, either with your data-to-backup, with missing cache, maybe never running forget/prune. But to judge if your backup run is too slow or not, you should fist post some outputs in combination with measurements.
There are actually various possibilities:
If you can mount, the repo, you can use this to restore a small amount of files (simply copy from the mounted dir)
A singe file can be dumped using dump, but this IMO is not the intended command for files, but for data streams like DB dumps you want to directly pipe to a restore-command.
The restore command supports include/exclude options, so you can specify which files you want to restore. I personally find this very clumsy - as alternative, you can wait for the <SNAPSHOT>:[PATH] syntax to be implemented (or use rustic which supports that syntax).
