Does resuming a interrupted backup also work with windows backing up over sftp?
The faq says it’s generally availible but I’m not sure why it’s actually not working in my setup.
simplified output of both runs:
// first backup (interrupted after a while)
scan finished in 1100.000s: 590000 files, 800.004 GiB
[1:14:00] 4.80% 20000 files 40.000 GiB, total 590000 files 800.000 GiB, 5 errors ETA 24:10:00
// second backup (as far as I see not resuming)
scan finished in 120.000s: 590033 files, 800.006 GiB
[2:16] 0.07% 3000 files 580.000 MiB, total 590033 files 800.006 GiB, 2 errors ETA 54:06:45
/C:/path/to/file/that/should-be-already-backed-up
/C:/path/to/other/file/that/should-be-already-backed-up
There are two phases to restic backing up your files. First it investigates the list of files and based on metadata determines which of these it hasn’t seen before and which of the ones it has seen before it does not need to back up because they haven’t changed in terms of metadata. After that it backs up the files that have changed or that are new.
To determine which files it has already seen before, restic must have at least one completed backup made already, which in your case it doesn’t. So since it cannot find a so called parent snapshot, in which it would have recorded the backed up files’ metadata, it has nothing to compare metadata with for the files you asked it to back up. Hence, it needs to rescan all of the files in order to determine what to actually send to the server. That’s why you are seeing it scanning the same amount of files the second time. It is also where it would have not spent time scanning files to be backed up, if it had a parent snapshot to compare with, in a sense “continuing” the backup “meta-data wise”.
However, when if in the first run you let it actually back up/send some files’ data, then that data will not be sent again. Restic will see that it has e.g. sent all of file X and parts of file Y and not re-send that data. This is where it will resume the backup “data-transfer wise”. This is always the case, regardless of whether there was a completed backup made before the current run or not.
To expand on what @rawtaz is saying, if there isn’t a prior completed backup then restic must re-read and re-hash all of the previously uploaded files in order to even determine that they have already been uploaded. So you’ll see a bunch of read I/O on your disk and a fair bit of CPU activity, but you should not see very much outbound network traffic until restic reaches data that it hasn’t already uploaded. Depending on the speed of your disk relative to how fast you can push data out on the network, this could mean that “resuming” an interrupted backup will take the same amount of time as a clean, non-resumed backup.
As far as I understand that, that means the scanning part and thats also not “my problem” because its only taking some minutes. It’s also understandably for me as written in the docs:
Note however that during the initial backup run and any re-tries, until there has been a first snapshot created for the backup set (list of files and directories to be backed up), restic will need to re-scan the files on disk as there is no parent snapshot to compare the filesystem with to determine which files have changed. This process should however be far quicker than the uploading, and it’s normal to see restic scan the files again when re-running the backup.
If I’m interpreting it correctly it sent already some data in the first (uncompleted) run but sent it again in the second try, as seen in a simplified example below:
// first run
[1:14:00] 4.80% 20000 files 40.000 GiB, total 590000 files 800.000 GiB, 5 errors ETA 24:10:00
// second run
[2:16] 0.07% 3000 files 580.000 MiB, total 590033 files 800.006 GiB, 2 errors ETA 54:06:45
(I forgot to put the first run info in the post, updated)
// second run
[2:16] 0.07% 3000 files 580.000 MiB, total 590033 files 800.006 GiB, 2 errors ETA 54:06:45
(I forgot to put the first run info in the post, updated)
I used a “not so elegant” solution for big initial backups: just first backups some of the subdirectories, so that it will complete quickly. Then when you backup the whole directory, restic will recognize the files that were already uploaded (it will hash them and recognize the same hash is already on the server) and so it will not upload them again. It is a bit inefficient because restic will hash the same files multiple times, but usually the time to upload a file is much higher than that of hashing it. Moreover on the server you will have a few weird snapshots that you will need to forget later, if you want to have a neat snapshots tree.
No, the scanning part is when restic determines which files need to have their content checked (which is always all files when there is no parent snapshot). The second phase, when the backup is actually constructed, involves hashing these files and uploading any chunks that do not already exist in the repository.
Moreover, you don’t gain much by doing this “subdirectory solution”. restic always saves pack files once they are full enough and index files if they are full enough or old enough. That means if you interrupt a large backup, you’ll lose the index contents of the files uploaded the last few minutes, but all other pack files are saved in the index and will be used if you rerun the backup (exactly like in your “subdirectory solution”). If you don’t want to lose that last minutes, you can even run rebuild-index (which has become very fast with 0.12.0) and all pack files will be used for subsequent backups. You just have to make sure that your don’t run a prune which would of course remove this so-far-not-needed data (this is the job of prune).
If you want to gain anything in your “resume backup”, you have to make sure that the files already contained in the repository will not be read and hashed again, see e.g. this thread and the discussion within:
