I was thinking… if I initiate a backup of lots of files, and the total size of the backup is large (1+ TB) and my Internet connection is slow (currently tracking about 950 Kbps upload), the initial backup may not finish before I need to do a restore. The problem is that, as I understand it, no snapshot file is created, so I cannot use restic to restore any of the data from an incomplete backup.
I know the proper solution is to start the backup earlier than it will be needed. But let’s suppose the initial backup gets to 75% to completion before I need to do a restore. Let’s suppose (for worst case) that I need a total restore. Darn, I can only get 75% of my data back – give or take the files’ blobs which uploaded only partially – but that’s still 75% which is better than zero.
Is it possible to restore from a partial snapshot? If not, it would be a nice feature to be able to tell restic to finsih the index for what has been uploaded, and then to create a snapshot file (maybe tag it as incomplete). This
way, we could presumably still access the 75% of the data that got stored and indexed.
You make an interesting point! I am note quite sure if it’s a good idea though.
If something like that would be implemented the user would have to know that the initial backup did not 100% succeed; there needs to be some sort of error message or warning letting the user know about this.
Plus there needs to be a lot of change to the logic on how snapshots are made - what happens when the user continues taking the snapshot? What happens if initial uploaded data has changed? Does that data need to be put in a new snapshot because the file has changed in the meantime? Do the “new”/not yet uploaded files get put in the original snapshot or a new one? Is there a need for an intermediate snapshot where all of the interrupted times a user tried to make a backup go into?
So I don’t know if this is a good idea actually. And I want to explain why:
If you think of the strategy 3-2-1 then a user would have a backup with the data on a storage solution in his/her home. Thus if an initial backup was to be made the first backup, IMHO, should be made to the storage which is the fastest - to the local storage solution whatever it may be. Then you can do a full restic check --read-data for example at no cost whatsoever.
Afterwards the user could then rclone the entire repository to a cloud solution provider of his/her choice - knowing that rclone for B2 does the SHA1 comparison making sure the data stored resembles the data uploaded and there is no corrupt data.
I believe this sort of user-error-handling is a little out of scope. As there are many ways how a user can work around that problem with other means that are already implemented and are not a huge overhead.
I don’t say this idea shouldn’t be added at all, but I worry about features being added and time being spent that could be spent wiser on core parts of restic. Feature creep is real.
Does that make sense? Feedback is highly appreciated!
I didn’t suggest this would happen automatically, rather, this would be a command the user themselves invokes manually if they need to do a restore on a snapshot that may have not completed (yet). Besides, after the snapshot is created, it would be tagged as incomplete, so it would show that in the snapshot listing.
Since this action would only be deliberate, I don’t know if the rest of your questions really apply.
Ok I can picture what you mean and how this could look like from a user perspective.
I think the rest of my question go the same way though - they are leaned towards how such a thing could be implemented and not confuse the user or make other functionalities redundant. Also none of them need a specific answer; it’s more of a thinking out loud process.
You can use any kind of transport, may it be Resilio Sync, rclone, sftp, ftp, you name it.
And by that I mean that you store the repository local (as in the 3-2-1 principle) and then copy that data to a cloud provider.
I do can see the benefit or resumable snapshots whatever the backend may be. I mean even Windows came around and made copy and past in the Explorer pause- and resume-able.
Maybe I am too afraid of feature creep…
But this doesn’t solve the problem of a long-running, incomplete initial (or any) backup. What if the laptop lid closes and the computer goes to sleep? What if the power goes out? And sometimes it just isn’t an option to back up locally.
I maintain several (and some large) open source projects – believe me, I’m wary of feature creep, but this is not feature creep. If restic cannot make a snapshot and restore from the data that is in the repo, what will? I guess we could write third-party restore tools, but restic packages are mostly internal, so they can’t be used as libraries easily, so we’d have to hand-roll the code ourselves from the design document (I think). It’s just a lot of work. It’s way less “creep-y” to just build it into restic (if I haven’t missed some way that this is already possible, of course).
Oh don’t get me wrong, as I said, I like the idea of restic being able to resume or pause a backup.
The reason I was so hesitant is not to shut you or the idea down but to try to talk about some ideas that came to my mind.
A good discussion is better than a post saying “Uh yeah that’s great! Can you ship that next week? Kthx”.
All the points you made are valid - especially the Laptop ones (Wifi cut off, backup started and you walk away with the laptop and then roaming doesn’t work properly, etc.).
So right now I am not sure what are the next big milestones other than the ones documented in the github - especially https://github.com/restic/restic/projects/3 (repo v2).
Maybe there are some ideas of making something like that possible in the future.
