Unable to access file open in another process

Getting Help
1

Hello

I am having an issue using Restic on Windows. Backups abort when it encounters a file that is open in another program/process (screendump below). I am backing up my ‘Documents’ Directory, which inevitably includes files which are currently open elsewhere (e.g. MS Outlook’s “sharepoint lists” file in the screendump below).

Ideal Behavior: Can restic somehow also access these files and back them up regardless, the same way notepad is able to open a file read-only even if it is open in Excel?

Great Behavior: When Restic encounters a file that’s open elsewhere, skip this file, but continue to back up the rest of the files

Current behavior: When restic encounters a file that’s open elsewhere, throw and error, save the snapshot for the files already backed up, but do not back up files not yet considered.

I would appreciate any help on this. Is there a switch on the backup command that I am missing?

C:\Users\JohnSmith>restic backup Documents
password is correct
using parent snapshot 299ae337
scan [C:\Users\JohnSmith\Documents]
scanned 94 directories, 370 files in 0:00
error for C:\Users\JohnSmith\Documents\SharePoint Lists.pst: chunker.Next: read \\?\C:\Users\JohnSmith\Documents\SharePoint Lists.pst: The process cannot
access the file because another process has locked a portion of the file.
warning for Documents\SharePoint Lists.pst: chunker.Next: read \\?\C:\Users\JohnSmith\Documents\SharePoint Lists.pst: The process cannot access the file because another process has locked a portion of the file.
[0:00] 9.38%  63.352 MiB / 675.618 MiB  463 / 464 items  1 errors  ETA 0:00
duration: 0:00
snapshot 6036a903 saved

C:\Users\JohnSmith>
2

This seems to be very similar to App hangs/fails to save open file during restic backup maybe check this as well https://github.com/restic/restic/issues/1347 as well as https://github.com/restic/restic/issues/340
Doesn’t really sound like it is a bug of restic, rather a limitation of Windows?!
As per: https://msdn.microsoft.com/en-us/library/ms837537.aspx

But anyone with more knowledge, please jump in!

3

Thanks for the response.

I don’t think it’s a bug in Restic (there is no one right answer for what should happen when encountering an error). However, the current behavior it is limiting the usefulness of Restic on Windows (at least for my use case).

I would prefer it if, on encountering a locked file, the backup skips this file and continues to backup all the other files in the directory. Is there a way to do this?

I had a look at the links you posted, but I was not able to figure out how to solve my problem based on their content.

4

Yes, I know and you are right that none of these links provide a “here is a fix”. My intent was more to show what the background to this could be or give some hint on where to look further to find a solution for your problem. As I said that I welcome more feedback from anyone.

5

Huh, so other processes can’t even read the file? Windows is so weird.

It does seem like restic should just log it and move on to the next file though in this case… :thinking:

6

It should work exactly this way, if not I consider this a bug! Restic should report errors, but still continue to save everything it can.

Which version of restic is that? Can you please retry with the latest master branch? We’ve recently exchanged the low-level code in this part of the program… You can find pre-compiled binaries here: restic beta releases (/)

7

Ideally, restic (or a wrapper around restic) would use the Windows Shadow Copy Service to allow access to these files. We already have a feature request for this. (mentioned by @moritzdietz above)

That github issue does appear to have some sample scripts showing how to write your own wrapper.

8

I was using v.0.8.3 before.

I now retried using restic 0.8.3 (v0.8.3-223-g2a672588) compiled with go1.10.2 on windows/amd64

I got a different error:

C:\Users\om90898>restic backup documents
repository 7203f76d opened successfully, password is correct
[0:02] 9.43%  363 files 63.777 MiB, total 372 files 676.170 MiB, 0 errors ETA 0:19
/documents/SharePoint Lists.pst
/documents/team planning.xlsx
/documents/~$ Protect - Policy Servicing Premium Impacts v0.9.4.docxpanic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x8 pc=0x76fd03]

goroutine 58 [running]:
github.com/restic/restic/internal/archiver.(*TreeSaver).save(0xc042115500, 0xb9fda0, 0xc042112740, 0xc042328e00, 0xa, 0xc042062140, 0xc042260000, 0x55, 0x55, 0xc044030640, ...)
        src/github.com/restic/restic/internal/archiver/tree_saver.go:107 +0x3b3
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc042115500, 0xb9fda0, 0xc042112740, 0xc042115518, 0xc042052900)
        src/github.com/restic/restic/internal/archiver/tree_saver.go:150 +0x117
created by github.com/restic/restic/internal/archiver.NewTreeSaver
        src/github.com/restic/restic/internal/archiver/tree_saver.go:65 +0x109
9

That has been reported as #1755 and will be fixed shortly in #1757.

1 Like
10

So I just tried it again on restic 0.9.2 (v0.9.2-62-gdf78896e) compiled with go1.10.2 on windows/amd64 and it still occurs. I think I saw you posting something about a Go release that should fix this kind of behavior?! I can’t seem to find the post or Github issue where this was discussed…
It might have been go 1.11.

11

Hm, can you describe the details? I can’t remember what we discussed, what happens when restic encounters a file which it cannot read? Does it abort? What does the snapshot contain? Can you paste the exact output? Is there a way to reproduce this for me?

12

Sure no problem - I am currently reproducing this here.

So this most often happens when you try to backup %Userprofile% which usually expands to C:\Users\<username\ and contains the %AppData% folder. In there I get the most errors when I try to run restic.

For example when I try to backup the AppData folder for Microsoft:

PS C:\WINDOWS\system32> C:\bin\restic.exe -r D:\test-repo backup C:\Users\Moritz\AppData\Local\Microsoft\
enter password for repository:
repository 66e6df8e opened successfully, password is correct
error: NodeFromFileInfo: Readlink: readlink \\?\C:\Users\Moritz\AppData\Local\Microsoft\Windows\INetCache\Content.IE5: Zugriff verweigert
error: open \\?\C:\Users\Moritz\AppData\Local\Microsoft\Windows\INetCache\Content.Word\~WRS{DEBA5E36-A9C4-445D-B2EE-94F26705F4CA}.tmp: Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen Prozess verwendet wird./Configuration/Config1.xml
error: NodeFromFileInfo: Readlink: readlink \\?\C:\Users\Moritz\AppData\Local\Microsoft\Windows\Temporary Internet Files: Zugriff verweigert
error: open \\?\C:\Users\Moritz\AppData\Local\Microsoft\Windows\UsrClass.dat: Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen Prozess verwendet wird./Users/Moritz/AppData/Local/Microsoft/Windows/Notifications/wpndatabase.db
error: open \\?\C:\Users\Moritz\AppData\Local\Microsoft\Windows\UsrClass.dat.LOG1: Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen Prozess verwendet wird.s/Moritz/AppData/Local/Microsoft/Windows/Notifications/wpndatabase.db
error: open \\?\C:\Users\Moritz\AppData\Local\Microsoft\Windows\UsrClass.dat.LOG2: Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen Prozess verwendet wird.s/Moritz/AppData/Local/Microsoft/Windows/Notifications/wpndatabase.db
error: open \\?\C:\Users\Moritz\AppData\Local\Microsoft\Windows\WebCache\V01.log: Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen Prozess verwendet wird.rs/Moritz/AppData/Local/Microsoft/Windows/UsrClass.dat{365a6924-4d37-11e8-a536-001a7dda7111}.TM.blf
error: open \\?\C:\Users\Moritz\AppData\Local\Microsoft\Windows\WebCache\V01tmp.log: Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen Prozess verwendet wird.Moritz/AppData/Local/Microsoft/Windows/WebCache/V010018D.log
error: open \\?\C:\Users\Moritz\AppData\Local\Microsoft\Windows\WebCache\WebCacheV01.dat: Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen Prozess verwendet wird.z/AppData/Local/Microsoft/Windows/WebCache/V010018D.log
error: open \\?\C:\Users\Moritz\AppData\Local\Microsoft\Windows\WebCache\WebCacheV01.jfm: Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen Prozess verwendet wird.z/AppData/Local/Microsoft/Windows/WebCache/V010018D.log
error: open \\?\C:\Users\Moritz\AppData\Local\Microsoft\Windows\WebCacheLock.dat: Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen Prozess verwendet wird.rs/Moritz/AppData/Local/Microsoft/Windows/WebCache/V010018D.log
error: NodeFromFileInfo: Readlink: readlink \\?\C:\Users\Moritz\AppData\Local\Microsoft\WindowsApps\Microsoft.MicrosoftEdge_8wekyb3d8bbwe\MicrosoftEdge.exe: Das System kann die angegebene Datei nicht finden.soft/Windows Sidebar/settings.ini
error: NodeFromFileInfo: Readlink: readlink \\?\C:\Users\Moritz\AppData\Local\Microsoft\WindowsApps\MicrosoftEdge.exe: Das System kann die angegebene Datei nicht finden.
error: NodeFromFileInfo: Readlink: readlink \\?\C:\Users\Moritz\AppData\Local\Microsoft\WindowsApps\PointPlanck.FileBot_49ex9gnthnt12\filebot.exe: Das System kann die angegebene Datei nicht finden.ocal/Microsoft/Windows Sidebar/settings.ini
error: NodeFromFileInfo: Readlink: readlink \\?\C:\Users\Moritz\AppData\Local\Microsoft\WindowsApps\filebot.exe: Das System kann die angegebene Datei nicht finden.

Files:         471 new,     0 changed,     0 unmodified
Dirs:            5 new,     0 changed,     0 unmodified
Added to the repo: 261.047 MiB

processed 471 files, 566.389 MiB in 0:11
snapshot ada6bdf2 saved
PS C:\WINDOWS\system32>

I also tried the .pst file example from the guy who initially created this thread.
This might have something to do with how other files open things?! I am not sure myself.
But here is a pst attached to Outlook + outlook being open:

PS C:\WINDOWS\system32> C:\bin\restic.exe -r D:\test-repo backup C:\Users\Moritz\Desktop\test.pst
enter password for repository:
repository 66e6df8e opened successfully, password is correct
error: read \\?\C:\Users\Moritz\Desktop\test.pst: Der Prozess kann nicht auf die Datei zugreifen, da ein anderer Prozess einen Teil der Datei gesperrt hat.

Files:           0 new,     0 changed,     0 unmodified
Dirs:            4 new,     0 changed,     0 unmodified
Added to the repo: 1.171 KiB

processed 0 files, 265.000 KiB in 0:03
snapshot b2921bfc saved
PS C:\WINDOWS\system32>
13

Just to clarify, it’s expected behavior that restic would report error and skip the file if it can’t be read. As long as restic continues to backup all other readable files, it’s not considered a bug.

You need to incorporate VSS to backup these unreadable files. This can be done using this PowerShell script.

14

Thanks for the confirmation, I wanted to make sure that a snapshot is created which contains at least all other files which could be read. That seems to be the case.

For errors, restic’s policy is trying to continue if possible, but report the error. Apart from VSS, I’m not sure what we (from restic’s side of view) can do here. I thought about maybe restic could retry reading the file, but for most of the files you mentioned it’s likely that on the next try, the file will still be locked, so it will only slow down the backup process…

15

Awesome! Thanks for getting back to me - now I can check this off in my mental list of things regarding restic on windows.