Initial backup so slow... 330h for 3 To

[Djo]

Following my problem here: My backup HDD is completely full. Now, how to prune?, I’ve decided to delete completely the repo and to re-create it.

Well, I didn’t expect this: 330 hours for approximately 3 TO to backup! Nearly 14 days non stop!
(2 To on the backup disk at the end)

The backup destination is a local USB 3.0 external disk WD Elements.
Windows 10.
My PC is in good shape…

Terribly slow…!
Fortunately backup times are much more acceptable after the initial backup.
But the initial backup times are not acceptable…

[Nev]

Probably not restic’s fault. Have a look at the specific drive being used in the WD Elements enclosure - is it an SMR drive?

These guys grind to a halt when copying many small files (which happens when restic first creates its repo). Actually, even a CMR drive will copy a few large files faster than the equivalent size in small files; it’s just the effect is greater with SMR.

This issue comes up quite regularly on the forum in relation to USB drives, and also cloud storage. There are ways you can improve the performance. Here’s the latest forum thread (this one is in relation to per-file cloud charges, but the workaround is the same):

Impose minimum file size

I was hoping to migrate my backup stack consisting of restic + rclone storing backups on B2 to Storj, however Storj imposes a $0.0000088 per file fee where B2 does not. I have in excess of 300k files and this would work out to $3 extra per month, while storing just shy of 2TB ($8/month at Storj) My question is, is there a way to force a minimum file size for the files produced by restic such that I could drastically decrease the amount of files stored? EDIT: slight correction, I have well ove…

[Eli6]

Based on my own backups, I expect max 24 hours for 3 TB, probably 15 hours.

[tomwaldnz]

Check if your antivirus product is scanning everything when restic is just checking timestamps on files to see if they need to be backed up. More detail in this thread.

[Djo]

Problem solved

What I didn’t say in my first post is that the USB disk was encrypted with VeraCrypt. And I remember that, in a way to quickly delete the repo, I did a quick format inside the encrypted volume of the disk with the format command of Windows. Well, maybe the filesystem didn’t like that and was altered (although no error was found).
More probably, the filesystem has gone corrupted because of the lack of free space (and the encryption didn’t help)

Anyway after a complete format of the disk then a re-creation of an encrypted VeraCrypt volume, the backup times has returned to normal. 10 hours to backup the 3 To!

Thanks all for the help.