Windows shadow copy snapshot: VSS UNEXPECTED PROVIDER ERROR?

jdiercks · March 9, 2021, 7:42pm

I’m getting an error when restic tries to create a VSS snapshot on Windows Server 2016:

c:\>restic backup --use-fs-snapshot c:\
...
restic.exe : error: failed to create snapshot for [c:\]: 
VSS error: AddToSnapshotSet() failed: 
VSS_E_UNEXPECTED_PROVIDER_ERROR (0x8004230f)
...

If I do a VSSADMIN LIST PROVIDERS, this provider shows up on the failing system but it is not present on a different system without the error:

Provider name: 'Hyper-V IC Software Shadow Copy Provider'
   Provider type: Software
Provider Id: {74600e39-7dc5-4567-a03b-f091d6c7b092}
   Version: 1.0.0.0

I’m told by the sysadmin that this provider is essential on this system, because we’re also doing daily snapshots to support “restore previous versions” in windows explorer.

So what else can I do in order to get restic to co-exist in this environment? Thanks.

–Jon

jdiercks · March 10, 2021, 1:03am

I ran another test and took a closer look at Event Viewer. The first errors that got thrown at the moment I begin the test are referencing Sentinel Agent Log VSS Writer, which would seem to point more directly to a conflict there. Details below.

This thread references some sort of VSS conflict between SentinelOne (S1) and regular Windows Server Backup. The workaround for them was to downgrade to an older S1 version, which I don’t think is an option for us, or wait for a fix in the “newest version”, not specified.

Anybody got any other ideas?

Volume Shadow Copy Service error: Unexpected error querying for the IVssWriterCallback interface.  hr = 0x80070005, Access is denied.
. This is often caused by incorrect security settings in either the writer or requestor process. 

Operation:
   Gathering Writer Data

Context:
   Writer Class Id: {90e12c7e-b9c2-4649-81a2-947d06dd5eba}
   Writer Name: Sentinel Agent Log VSS Writer
   Writer Instance Name: C:\ProgramData\Sentinel\logs\SentinelOne_7388.binlog
   Writer Instance ID: {cf3b2a43-129a-47da-836d-dc08c6432059}


Volume Shadow Copy Service error: Unexpected error querying for the IVssWriterCallback interface.  hr = 0x80070005, Access is denied.
. This is often caused by incorrect security settings in either the writer or requestor process. 

Operation:
   Gathering Writer Data

Context:
   Writer Class Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Name: System Writer
   Writer Instance ID: {1fc2a385-de5f-4420-aa2e-90d647016e57}


Volume Shadow Copy Service error: Unexpected error querying for the IVssWriterCallback interface.  hr = 0x80070005, Access is denied.
. This is often caused by incorrect security settings in either the writer or requestor process. 

Operation:
   Gathering Writer Data

Context:
   Writer Class Id: {7e47b561-971a-46e6-96b9-696eeaa53b2a}
   Writer Name: MSMQ Writer (MSMQ)
   Writer Instance Name: MSMQ Writer (MSMQ)
   Writer Instance ID: {f2d68a0b-7dd2-4953-b102-8e0a116a0e25}


Volume Shadow Copy Service error: A critical component required by the Volume Shadow Copy service is not registered. This might happened if an error occurred during Windows setup or during installation of a Shadow Copy provider. The error returned from CoCreateInstance on class with CLSID {3e02620c-e180-44f3-b154-2473646e4cb8} and Name SW_PROV is [0x80040154, Class not registered
]. 

Operation:
   Obtain a callable interface for this provider
   List interfaces for all providers supporting this context
   Check If Volume Is Supported by Provider

Context:
   Provider ID: {74600e39-7dc5-4567-a03b-f091d6c7b092}
   Class ID: {3e02620c-e180-44f3-b154-2473646e4cb8}
   Snapshot Context: 0
   Snapshot Context: 0
   Execution Context: Coordinator
   Provider ID: {00000000-0000-0000-0000-000000000000}
   Volume Name: c:\


Volume Shadow Copy Service error: Error creating the Shadow Copy Provider COM class with CLSID {3e02620c-e180-44f3-b154-2473646e4cb8} [0x80040154, Class not registered
]. 

Operation:
   Obtain a callable interface for this provider
   List interfaces for all providers supporting this context
   Check If Volume Is Supported by Provider

Context:
   Provider ID: {74600e39-7dc5-4567-a03b-f091d6c7b092}
   Class ID: {3e02620c-e180-44f3-b154-2473646e4cb8}
   Snapshot Context: 0
   Snapshot Context: 0
   Execution Context: Coordinator
   Provider ID: {00000000-0000-0000-0000-000000000000}
   Volume Name: c:\


Volume Shadow Copy Service error: A critical component required by the Volume Shadow Copy service is not registered. This might happened if an error occurred during Windows setup or during installation of a Shadow Copy provider. The error returned from CoCreateInstance on class with CLSID {3e02620c-e180-44f3-b154-2473646e4cb8} and Name SW_PROV is [0x80040154, Class not registered
]. 

Operation:
   Obtain a callable interface for this provider
   List interfaces for all providers supporting this context
   Check If Volume Is Supported by Provider
   Add a Volume to a Shadow Copy Set

Context:
   Provider ID: {74600e39-7dc5-4567-a03b-f091d6c7b092}
   Class ID: {3e02620c-e180-44f3-b154-2473646e4cb8}
   Snapshot Context: 4194304
   Snapshot Context: 4194304
   Execution Context: Coordinator
   Provider ID: {00000000-0000-0000-0000-000000000000}
   Volume Name: c:\
   Execution Context: Coordinator

Volume Shadow Copy Service error: Error creating the Shadow Copy Provider COM class with CLSID {3e02620c-e180-44f3-b154-2473646e4cb8} [0x80040154, Class not registered
]. 

Operation:
   Obtain a callable interface for this provider
   List interfaces for all providers supporting this context
   Check If Volume Is Supported by Provider
   Add a Volume to a Shadow Copy Set

Context:
   Provider ID: {74600e39-7dc5-4567-a03b-f091d6c7b092}
   Class ID: {3e02620c-e180-44f3-b154-2473646e4cb8}
   Snapshot Context: 4194304
   Snapshot Context: 4194304
   Execution Context: Coordinator
   Provider ID: {00000000-0000-0000-0000-000000000000}
   Volume Name: c:\
   Execution Context: Coordinator


Volume Shadow Copy Service error: A critical component required by the Volume Shadow Copy service is not registered. This might happened if an error occurred during Windows setup or during installation of a Shadow Copy provider. The error returned from CoCreateInstance on class with CLSID {3e02620c-e180-44f3-b154-2473646e4cb8} and Name SW_PROV is [0x80040154, Class not registered
]. 

Operation:
   Obtain a callable interface for this provider
   Add a Volume to a Shadow Copy Set

Context:
   Provider ID: {74600e39-7dc5-4567-a03b-f091d6c7b092}
   Class ID: {3e02620c-e180-44f3-b154-2473646e4cb8}
   Snapshot Context: 4194304
   Execution Context: Coordinator


Volume Shadow Copy Service error: Error creating the Shadow Copy Provider COM class with CLSID {3e02620c-e180-44f3-b154-2473646e4cb8} [0x80040154, Class not registered
]. 

Operation:
   Obtain a callable interface for this provider
   Add a Volume to a Shadow Copy Set

Context:
   Provider ID: {74600e39-7dc5-4567-a03b-f091d6c7b092}
   Class ID: {3e02620c-e180-44f3-b154-2473646e4cb8}
   Snapshot Context: 4194304
   Execution Context: Coordinator


Volume Shadow Copy Service error: Unexpected error querying for the IVssWriterCallback interface.  hr = 0x80070005, Access is denied.
. This is often caused by incorrect security settings in either the writer or requestor process. 

Operation:
   Gathering Writer Data

Context:
   Writer Class Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Name: System Writer
   Writer Instance ID: {1fc2a385-de5f-4420-aa2e-90d647016e57}


Volume Shadow Copy Service error: Unexpected error querying for the IVssWriterCallback interface.  hr = 0x80070005, Access is denied.
. This is often caused by incorrect security settings in either the writer or requestor process. 

Operation:
   Gathering Writer Data

Context:
   Writer Class Id: {90e12c7e-b9c2-4649-81a2-947d06dd5eba}
   Writer Name: Sentinel Agent Log VSS Writer
   Writer Instance Name: C:\ProgramData\Sentinel\logs\SentinelOne_7388.binlog
   Writer Instance ID: {cf3b2a43-129a-47da-836d-dc08c6432059}


Volume Shadow Copy Service error: Unexpected error querying for the IVssWriterCallback interface.  hr = 0x80070005, Access is denied.
. This is often caused by incorrect security settings in either the writer or requestor process. 

Operation:
   Gathering Writer Data

Context:
   Writer Class Id: {7e47b561-971a-46e6-96b9-696eeaa53b2a}
   Writer Name: MSMQ Writer (MSMQ)
   Writer Instance Name: MSMQ Writer (MSMQ)
   Writer Instance ID: {f2d68a0b-7dd2-4953-b102-8e0a116a0e25}


Volume Shadow Copy Service error: A critical component required by the Volume Shadow Copy service is not registered. This might happened if an error occurred during Windows setup or during installation of a Shadow Copy provider. The error returned from CoCreateInstance on class with CLSID {3e02620c-e180-44f3-b154-2473646e4cb8} and Name SW_PROV is [0x80040154, Class not registered
]. 

Operation:
   Obtain a callable interface for this provider
   List interfaces for all providers supporting this context
   Check If Volume Is Supported by Provider

Context:
   Provider ID: {74600e39-7dc5-4567-a03b-f091d6c7b092}
   Class ID: {3e02620c-e180-44f3-b154-2473646e4cb8}
   Snapshot Context: 0
   Snapshot Context: 0
   Execution Context: Coordinator
   Provider ID: {00000000-0000-0000-0000-000000000000}
   Volume Name: n:\


Volume Shadow Copy Service error: Error creating the Shadow Copy Provider COM class with CLSID {3e02620c-e180-44f3-b154-2473646e4cb8} [0x80040154, Class not registered
]. 

Operation:
   Obtain a callable interface for this provider
   List interfaces for all providers supporting this context
   Check If Volume Is Supported by Provider

Context:
   Provider ID: {74600e39-7dc5-4567-a03b-f091d6c7b092}
   Class ID: {3e02620c-e180-44f3-b154-2473646e4cb8}
   Snapshot Context: 0
   Snapshot Context: 0
   Execution Context: Coordinator
   Provider ID: {00000000-0000-0000-0000-000000000000}
   Volume Name: n:\

punchcard · March 11, 2021, 12:27am

I am not likely qualified to give you great answer but here goes at a general suggestion: There seems to be someone else who is doing backups. What is the reason for duplicating the effort? There can be lots of good reasons like they charge for recovery or are not timely or this is a testing environment and restores are done multiple times which do not match up to when the other group does backups. Do you need to backup the whole C: drive or can you just backup C:\Users or a specific list of directories (and exported registry)? Is the id which you are using an administrator on the machine? I’m not sure if administration priv is required.

jdiercks · March 11, 2021, 12:36am

Thanks for the reply. Those are all questions worth considering…

Yes, this will be a “redundant” backup; we need one that is completely under our control, just in case the hosting company is ever unable to provide us a restore from their normal backups in a timely manner.
We will be backing up specific folders, but we still want the benefits of VSS snapshots so that files will not be prevented from being backed up if they are open.
Yes, the user executing restic is a domain admin.

jdiercks · March 18, 2021, 11:21pm

Still trying to track this down. Testing fails consistently, but the reported error alternates randomly between these two:

VSS_E_UNEXPECTED_PROVIDER_ERROR
VSS_E_SNAPSHOT_SET_IN_PROGRESS

I’ve tried explicitly granting myself VSS permission via reg key:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\VssAccessControl

And also tried running restic as a scheduled task logged on as SYSTEM instead of my normal domain admin account. No changes in results.

“VSSADMIN LIST WRITERS” shows several writers registered, but status is all “stable” with no errors.

@fgma do you have any other ideas?

jdiercks · March 19, 2021, 12:06am

update: there is a pattern…

VSS_E_UNEXPECTED_PROVIDER_ERROR appears once, first, and then VSS_E_SNAPSHOT_SET_IN_PROGRESS appears consistently after that for a few minutes, presumably until the first attempt fully times out.

I can successfully do a “VSSADMIN CREATE SHADOW /For=C:”, as long as there hasn’t been a VSS_E_UNEXPECTED_PROVIDER_ERROR thrown by restic recently. So if VSSADMIN can create a shadow, why can’t restic do it when logged on as the same user?

fgma · March 19, 2021, 12:08pm

I’ve just did a bit of research. The pattern you mentioned is plausible. When creating VSS_E_UNEXPECTED_PROVIDER_ERROR a snapshot will not be cleaned up correctly so it stays there for a few minutes, restart of VSS service or a reboot of the system.

You mentioned you can create a snapshot manually using vssadmin. Can you check after doing so which provider created this snapshot (“vssadmin list shadows”)? In my case this is always “Provider: ‘Microsoft Software Shadow Copy provider 1.0’”. as this is the only provider on my system.

jdiercks · March 19, 2021, 3:26pm

Thanks for looking into this with me!

Yes, same provider for the manual vssadmin snapshot: ‘Microsoft Software Shadow Copy provider 1.0’

But I do have multiple providers on the system:

C:\gcpa>vssadmin list providers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2013 Microsoft Corp.

Provider name: 'Microsoft iSCSI Target VSS Hardware Provider'
   Provider type: Hardware
   Provider Id: {3f900f90-00e9-440e-873a-96ca5eb079e5}
   Version: 10.0.14393.4169

Provider name: 'Hyper-V IC Software Shadow Copy Provider'
   Provider type: Software
   Provider Id: {74600e39-7dc5-4567-a03b-f091d6c7b092}
   Version: 1.0.0.0

Provider name: 'Microsoft File Share Shadow Copy provider'
   Provider type: Fileshare
   Provider Id: {89300202-3cec-4981-9171-19f59559e0f2}
   Version: 1.0.0.1

Provider name: 'Microsoft Software Shadow Copy provider 1.0'
   Provider type: System
   Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}
   Version: 1.0.0.7

jdiercks · March 19, 2021, 9:26pm

@fgma the DLL call that is failing is AddToSnapshotSet.

According to this:

There should be 3 parameters on the call:

HRESULT AddToSnapshotSet(
  VSS_PWSZ pwszVolumeName,
  VSS_ID   ProviderId,
  VSS_ID   *pidSnapshot
);

And I see that the restic call to the DLL looks like this, in vss_windows.go…

		result, _, _ = syscall.Syscall6(vss.getVTable().addToSnapshotSet, 4,
			uintptr(unsafe.Pointer(vss)), uintptr(unsafe.Pointer(volumeNamePointer)),
			uintptr(unsafe.Pointer(ole.IID_NULL)), uintptr(unsafe.Pointer(idSnapshot)), 0, 0)

Is the IID_NULL what’s beeing passed as ProviderId? The MS docs do say:

If ProviderId is GUID_NULL, the default provider is selected according to the following algorithm:

If any hardware provider supports the given volume or remote file share, that provider is selected.

If there is no hardware provider available, if any software provider supports the given volume, it is selected.

If there is no hardware provider or software provider available, the system provider is selected. (There is only one preinstalled system provider, which must support all nonremovable local volumes.)

Maybe we need a way to explicitly specify a provider via a restic --option ?

Or am I completely barking up the wrong tree? I confess I’m a little out of my depth.

jdiercks · March 19, 2021, 10:13pm

It just keeps getting weirder…

I have now tested on 3 different servers, all very similarly configured except that they all have different combinations of VSS providers. Only one is failing (and of course that’s the one I need to have working).

What’s really strange is that it’s only failing on the server that has both the iSCSI VSS Hardware Provider and the Hyper-V IC Software Provider. The other two servers have one or the other, but not both, and neither of those get the error.

__________________________________________________
server 4683:

C:\>vssadmin list providers | find "name:"
Provider name: 'Microsoft iSCSI Target VSS Hardware Provider'
Provider name: 'Microsoft File Share Shadow Copy provider'
Provider name: 'Microsoft Software Shadow Copy provider 1.0'

C:\>restic backup --verbose --use-fs-snapshot ...
...
successfully created snapshot for [c:\]

__________________________________________________
server 4690:

C:\>vssadmin list providers | find "name:"
Provider name: 'Hyper-V IC Software Shadow Copy Provider'
Provider name: 'Microsoft File Share Shadow Copy provider'
Provider name: 'Microsoft Software Shadow Copy provider 1.0'

C:\>restic backup --verbose --use-fs-snapshot ...
...
successfully created snapshot for [c:\]

__________________________________________________
server 4381:

C:\>vssadmin list providers | find "name:"
Provider name: 'Microsoft iSCSI Target VSS Hardware Provider'
Provider name: 'Hyper-V IC Software Shadow Copy Provider'
Provider name: 'Microsoft File Share Shadow Copy provider'
Provider name: 'Microsoft Software Shadow Copy provider 1.0'

C:\>restic backup --verbose --use-fs-snapshot ...
...
restic.exe : error: failed to create snapshot for [c:\]: VSS error: AddToSnapshotSet() failed: VSS_E_UNEXPECTED_PROVIDER_ERROR (0x8004230f)

fgma · March 19, 2021, 10:37pm

You just posted what I wanted to post.

Restic will always use the default provider. I wonder about vssadmin.You can’t set the provider to use. Maybe it always uses the system provider so it will succeed but restic will choose different one.

Could you verify this based on Arcserve Support? I can’t find any other tool which allows to specify the provider when creating a snapshot.

I’ve found articles about commercial backup tools having the same problem. Most of them tell you to remove the other VSS provider. Acronis seems to have a settings to always use the system provider. I wonder if this is a good idea or if it will lead to other problems where it may be necessary to use a special provider to get the correct result.

jdiercks · March 19, 2021, 10:59pm

That Arcserv link looks promising - a regkey that forces using the default. I’ll try that this weekend when I can schedule a reboot. Thanks!

jdiercks · March 21, 2021, 1:17am

Still waiting to schedule a reboot, but I think I also want to try making a test build of restic that explicitly specifies the standard MS Provider (ID = “{b5946137-7b9f-4925-af80-51abd60b20d5}”).

I’m new to golang, but I have successfully set up a build environment in an LXD container. What would be the correct syntax I should use to replace:

uintptr(unsafe.Pointer(ole.IID_NULL))

with the constant, {b5946137-7b9f-4925-af80-51abd60b20d5}, for testing purposes? Thanks.

jdiercks · March 21, 2021, 4:18am

Think I found what I needed:

uintptr(unsafe.Pointer(ole.NewGUID("{b5946137-7b9f-4925-af80-51abd60b20d5}")))

… and it seems to be working!
More testing to do, but so far cautiously optimistic.

Ultimately the best thing to do will probably be to add an option like ‘–option vss.provider=b5946137-7b9f-4925-af80-51abd60b20d5’ so that any desired provider can be explicitly specified. Might have to learn some more golang and actually roll my own PR for this.

fgma · March 22, 2021, 6:26pm

You should probably open up an issue on github to discuss this further.

If we end up with an option to specify the provider manually its not that much work to implement it. I would do so but first I’d like to get some feedback from other people involved.

jdiercks · March 22, 2021, 6:47pm

Thanks. Made an initial inquiry here in this existing PR:

Schroedingers-Cat · December 28, 2022, 2:57pm

I’m seeing the error from post #2 in the Windows 10 event log whenever I create a restic snapshot with the “–use-fs-snapshot” option. It’s in German on my system, but it’s the same text only with different GUIDs:

Volumeschattenkopie-Dienstfehler: Beim Abfragen nach der Schnittstelle "IVssWriterCallback" ist ein unerwarteter Fehler aufgetreten. hr = 0x80070005, Zugriff verweigert
. Die Ursache hierfür ist oft eine falsche Sicherheitseinstellung im Schreib- oder Anfrageprozess. 

Vorgang:
   Generatordaten werden gesammelt

Kontext:
   Generatorklassen-ID: {e8132975-6f93-4464-a53e-1050253ae220}
   Generatorname: System Writer
   Generatorinstanz-ID: {77b13520-6bd0-45d2-a85c-ab7ab8d8b9a0}

Restic itself doesn’t report any issue, I only have these in the Windows event view.
There’s only a single VSS provider on my system:

vssadmin list providers
vssadmin 1.1 - Verwaltungsbefehlszeilenprogramm des Volumeschattenkopie-Dienstes
(C) Copyright 2001-2013 Microsoft Corp.

Anbietername: "Microsoft Software Shadow Copy provider 1.0"
   Anbietertyp: System
   Anbieterkennung: {b5946137-7b9f-4925-af80-51abd60b20d5}
   Version: 1.0.0.7

I don’t use Windows Server, just plain Windows 10 Pro. I already tried setting the VssAccessControl DWORDS in the registry to the Administrator and my local account, but that didn’t help. Any idea what that could be?