Find progress with initial backup?

I’ve got restic snapshotting nicely to my local file system.

Yesterday I worked out how to do snapshotting to Google Drive using rclone (on Windows). This worked without any problems with a “toy” backup.

The trouble is that my main valuable directory, My Documents, has about 5 GB in it, and I have a very slow upload bandwidth. And I mean really slow, like 1 Mbps.

This hopefully will be just about enough when the repo is already in place, as occasional files are added, deleted or modified. But getting the original 5GB repo (first snapshot) in place is naturally proving challenging.

I read here that restic backup is able to resume snapshotting if it gets cut off mid-flow, which is great. But I have no idea what percentage has so far been processed, or indeed whether this is practical to attempt this at all, at 1 Mbps.

Is there any command that can help out with this (i.e. comparing what half-built snapshot might be in place with the source local directory, and give a “percentage done” type of figure)?

In this post I tried to explain how resuming of interrupted backups currently works in restic:

A subsequent post in this thread recommends splitting the initial backup into smaller parts which might help in your case aswell.

@alexweiss made a PR to improve resumption of interrupted backups. Note that this PR hasn’t been merged yet and might have bugs, but you could give it a shot.

Thanks.

It occurred to me in the mean time to do a snapshot locally, and then upload that repo “manually” overnight to GDrive.

That looks like quite a big change in that PR: maybe I’ll leave it to the experts to give it the stamp of approval and merge it! (Also it doesn’t give me what I want, i.e. a “% of job done”…)

Yes, that’s a good solution as well.

That’s true but giving estimates with restic is difficult as restic doesn’t know beforehand how it has to upload because this depends on deduplication.

If I have done the math correctly your initial snaphot should be done in less than 12 hours (unless it’s interrupted).

Sorry, I got this all wrong. I just did a new snapshot at the command line (I’ve got used to doing everything through Python scripting), and of course this does show percentage done, and even an “ETA”.

In a Python scripting context I’m currently just ignoring all this output until a JSON object with message_type “summary” is delivered. But this is all I was asking about really: a ballpark figure based on how much volume has been processed so far and how much remains.

1 Like

5GB takes about 11 hours to transmit at 1Mbps, which seems like a reasonable amount of time to me. 5TB on the other hand would take more than a year. You could exclude the very large folders initially, do the first backup and let it complete, then add the larger folders over time.

What I do in some cases is run the restic backup locally then use external software to do the upload. In my case this is the S3 CLI but you could put it on a disk and copy it manually if your destination can handle this.