Hi Folks. First of all, I want to mention how excited I am to discover restic. From my initial experiments it seems like a great backup tool. The de-duplication, cloud storage backends, and cross-platform support are just what I’m looking for.
However, I’m running into an issue on Windows 10. I’m trying to backup approximately 250GB to a local restic repository (eventually it will be stored on Backblaze B2), and I’m seeing errors manipulating temp files. I thought I was perhaps running into drive space issues, but I’ve tried moving the temp folder to a drive with more space but still encountered the same error. I can sync files to a Linux server and backup from there, but I’d rather avoid the extra complexity. Am I doing something wrong? The full output is shown below.
D:\>restic backup brett\OneDrive\Pictures
password is correct
scan [D:\brett\OneDrive\Pictures]
scanned 628 directories, 38514 files in 0:14
[1:02:15] 52.01% 34.894 MiB/s 127.275 GiB / 244.735 GiB 29325 / 39142 items 0 errors ETA 57:26
error while saving data to the repo: remove d:\temp\restic-temp-pack-843627095: The process cannot access the file because it is being used by another process.
Remove
github.com/restic/restic/internal/repository.(*Repository).savePacker
src/github.com/restic/restic/internal/repository/packer_manager.go:136
github.com/restic/restic/internal/repository.(*Repository).SaveAndEncrypt
src/github.com/restic/restic/internal/repository/repository.go:253
github.com/restic/restic/internal/repository.(*Repository).SaveBlob
src/github.com/restic/restic/internal/repository/repository.go:618
github.com/restic/restic/internal/archiver.(*Archiver).Save
src/github.com/restic/restic/internal/archiver/archiver.go:106
github.com/restic/restic/internal/archiver.(*Archiver).saveChunk
src/github.com/restic/restic/internal/archiver/archiver.go:171
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:2337
panic: Remove: remove d:\temp\restic-temp-pack-843627095: The process cannot access the file because it is being used by another process.
goroutine 112901 [running]:
github.com/restic/restic/internal/archiver.(*Archiver).saveChunk(0xc042172000, 0xdee880, 0xc0420312c0, 0x35b1e7b, 0x1180b4, 0x60e9d1c700000, 0xc04a94a000, 0x1180b4, 0x13a000, 0xc042086420, ...)
src/github.com/restic/restic/internal/archiver/archiver.go:176 +0x422
created by github.com/restic/restic/internal/archiver.(*Archiver).SaveFile
src/github.com/restic/restic/internal/archiver/archiver.go:250 +0x3d7
I suggest you use e.g. Process Explorer from http://www.sysinternals.com to investigate what processes have that file open. Perhaps you can spot something relevant. You can have it record for you and review the result later.
This feels like restic is saving its own cache… but from the paths you’re using it does not look that way.
The error is caused by restic trying to remove its own temp files, but another process is using it. Which is strange, and I’ve only seen this when users try to backup a path which includes the tempdir itself (so restic is both trying to remove the tempfile and simultaneously reading/archiving it).
Is there something else accessing the temp files? A virus scanner or so?
I’ll try it out. It’s been a long time since I used Process Explorer, can it record all activity for a directory? If it can I might be able to narrow it down by giving restic its own temp folder and looking for any non-restic processes touching it.
Yep, virus scanner might be a possibility. I’m just using the inbuilt windows scanner/malware functionality. Perhaps it’s causing the issue. I’ll try disabling it.
I’ll try out the suggestions this evening (in about 12 hours).
Hm, I’m not sure what to do with this error from the restic side of things. On the one hand, aborting the backup because at temp file could not be removed feels wrong. On the other hand, when that happens several times, the user’s hard disc will fill up. Hm. Thoughts on that?
Option 1, fatal error: Doesn’t complete the backup, when unattended might cause a lot of failed backups before someone notices. Not great.
Option 2, continuing work but emitting a warning along with a notice that it might cause disk space to fill up: Better solution. Backups will complete so there’s less chance of critical problems in the short run.
In both cases people should notice that something is wrong, otherwise they set up their unattended backups/logging improperly, but risking the disk becoming full is IMO better than knowing that backups will not complete. People might not have time to address the latter right away, but might be fine knowing that they do have space on the disk to accomodate a number of additional runs with non-removed temp files.
PS: It’s not uncommon for various software and other factors to cause temp files being left behind, both in general and more specifically due to software prematurely terminating.
I’ve configured Process Monitor (not Process Explorer) from Sysinternals which allows you to filter for events on a directory. I’ve found two possible culprits.
C:\ProgramData\Microsoft\Windows Defender\Platform\4.12.17007.17123-0\MsMpEng.exe This is Microsoft’s anti malware real-time scanner.
C:\Program Files (x86)\IDriveWindows\id_service.exe I’ve been experimenting with the IDrive backup.
Based on the events, I believe the anti-malware executable is the more likely problem. I think it’s the only one actually opening the files, the IDrive client appears to be scanning file names.
I’ll try disabling the real-time file monitoring temporarily and see if it fixes the problem.
Oops, I thought I’d already provided an update, apparently I didn’t hit send.
The backups are now completing successfully with the real-time file monitoring disabled. It will possibly be okay now that backups are fast and incremental.
As for how to fix this long term, it does seem like something that will occur to other Windows users. Of the ideas discussed my vote would be to warn and continue if temp files cannot be removed. Perhaps the backup process could still return a (specific?) non-zero error code so that it can be investigated. Alternatively, is it possible to simply retry the delete? I suspect the virus scanner will only lock the file temporarily.