Need help w/ slooow (also large) Backup from Win10 to B2

Getting Help
1

As the title says I need help troubleshooting my setup with restic.

lt;dr: I’m trying to back-up a ~200GB Folder with mostly small files from Win10 to a Backblaze B2 bucket. I don’t like it when my ETA tries to climb into the triple digit hours.
It’s possible I’m limited by upload speed, but even then I simply can’t have restic run continuously for a day and a half.
How do I troubleshoot the problem and do you know a workaround?

Long version
Due to circumstances™, I have a lot of duplicate data I want to do some spring cleaning on, also I’d like to replace/upgrade my drives.
Before that, and once again I’ve decided that now is the time I fix my backups, but it’s not going well.

My original plan from several years ago was to have a semi-weekly Scheduled Task to back-up my computer to a B2 Bucket in the bakground.
(And another to prompt me to connect an external hdd and back-up to it daily.
I quickly abandoned that plan and started using a dedicated internal hdd for a partial backup instead. But that drive is old and loud and needs replacing. (Maintenance is hard ;-; ))

I never got the Task to actually run when set to background, but the more pressing issue is that backing up to B2 takes incredibly long. I have never finished a full backup. I know I’m backing up a lot of data without a parent snapshot, but with restic running, my PC is basically idle. I’d expect Upload to be constantly at the usual 20-25MBit/s, but it often dips to 30-60% of that.

I need a road to the goal of “I have a backup”. Pls help?

What I’ve tried:

  • Deactivate Microsoft Defender for a minute
  • Exclude the path to the restic executable from Defender
  • Exclude restic.exe from Defender
  • speedtest_net

Considerations:

  • Possibly upload speed is a limiting factor?
  • Maybe I can mail them a thumbdrive lol
  • Try back-up from ubuntu live image to see if speed is better
  • Maybe Defender settings are only effective after reboot?
  • clone the drive and hope for the best

More details:
Windows 10 Pro 64Bit Version 21H1 (Build 19043.1237)
on decent-ish hardware
restic 0.12.0 compiled with go1.15.8 on windows/amd64
installed via scoop with global option (C:\ProgramData…)
Folder on secondary drive, WD Blue 1TB, NTFS conected via SATA

2

First off, try using the latest master build at beta.restic.net and verify that it’s still slow.

Second I would want to know more about the output - can you paste the output of the backup run (including its command)?

3

I tried it with the latest beta, and found strong evidence that setting the Defender exemption is effective immediately (and makes a world of difference).

I think the ETA is much lower now? It’s still almost constantly climbing during the backup, but at a rate that may be explained with restic breezing through already backed-up files for the first 30-40 minutes.
[2:06:03] 35.27% 125537 files 68.367 GiB, total 189922 files 193.831 GiB, 0 errors ETA 3:51:19

This is what the network activity looks like right now:

grafik
grafik696×574 23.2 KB

And the core parts of my script:

$env:RESTIC_REPOSITORY=""
$env:RESTIC_PASSWORD=""
$env:B2_ACCOUNT_ID=""
$env:B2_ACCOUNT_KEY=""

$logFile = "./restic-" + (Get-Date -Format FileDate) + ".log"
Start-Transcript -Append -Path $logFile

.\restic_v0.12.1-98-ga5e103a2_windows_amd64.exe backup E:\Folder\ --exclude-file files_to_exclude.txt --exclude-caches

Stop-Transcript
4

Alright, it’s going much better now. Thanks for the reply! (Was this the output you requested btw?)

However on successive incomplete runs, restic still takes quite some time (currently ~40 minutes) before reaching files that haven’t been backed up. By then its progress is stated as >35%, (and then the ETA starts ticking up.)

I bet that’s a case the formula calculating the ETA assumes “changes” to the folder being backed up are distributed uniformly. I bet that could be improved without much trouble… :thinking:

I think that means a backup-run after a 99.9% run would take about 2h to complete.
I sure hope there are optimizations that grip once the backup has a parent, but 2h would be acceptable.

Is there something I can do to speed up these first ones? Basically telling restic to reuse yesterday’s scan first, and skipping ahead to traverse new and uncharted parts of the dir before traversing the same (unchanged) part of the file system ten times?

5

I think this isn’t something buggy or over-complicated, but rather just a case of some resources being less performant than expected. I suggest you simply finish the first backup such that restic gets a parent snapshot it can use for detecting changes, until then it needs to scan all files every time. So just finish it and see how subsequent backup runs (after the first completed one) performs.

No, of course not. If someone asks for thing X and you provide a part of X, then that is not what was asked for :wink: I don’t think it’s relevant anymore though.

6

Eventually I will …

But I’ll probably have to manually split the data into small pieces to back it up, because re-scanning everything that is alphabetically before where I left off is taking frustratingly long

7

Can you not let it sit there and run until it’s finished? Yeah, if you cannot do that, don’t hesistate to split it up a bit.