Errors restoring with restic on Windows Server + S3

Getting Help
1

Hello.
I have a Windows Server 2019 to backup (local 1,4 TB .vhdx file created with Windows Server Backup) and S3 backend.
I run backup command and it runs without any issue:

restic -r s3:https://s3.somecloudservice.com:443/backup --password-file C:\key.txt backup D:\WindowsImageBackup

Then I run restore and it shows errors during process:

C:\>restic -r s3:https://s3.somecloudservice.com:443/backup --password-file C:\key.txt restore b2058b61 --target D:\
repository c6c33138 opened (version 2, compression level auto)
[0:09] 100.00%  88 / 88 index files loaded
restoring <Snapshot b2058b61 of [D:\WindowsImageBackup] at 2023-11-02 17:28:17.3508685 +0300 MSK by addomain\user@server> to D:\
Load(<data/9084af745f>, 15123823, 1642536) returned error, retrying after 566.345324ms: ReadFull: read tcp 10.11.12.13:24815->1.2.3.4:443: wsarecv: An existing connection was forcibly closed by the remote host.
Load(<data/9084af745f>, 15123823, 1642536) returned error, retrying after 566.345324ms: ReadFull: read tcp 10.11.12.13:24815->1.2.3.4:443: wsarecv: An existing connection was forcibly closed by the remote host.
Load(<data/eb55b42404>, 16900531, 0) returned error, retrying after 442.352923ms: ReadFull: read tcp 10.11.12.13:24819->1.2.3.4:443: wsarecv: An existing connection was forcibly closed by the remote host.
ignoring error for \D\WindowsImageBackup\server\Backup 2023-11-01 071059\31845d83-74d5-4bce-a53e-a0356e5bf4f2.vhdx: StreamPack: ReadFull: read tcp 10.11.12.13:24817->1.2.3.4:443: wsarecv: An existing connection was forcibly closed by the remote host.
ignoring error for \D\WindowsImageBackup\server\Backup 2023-11-01 071059\31845d83-74d5-4bce-a53e-a0356e5bf4f2.vhdx: StreamPack: ReadFull: read tcp 10.11.12.13:37424->1.2.3.4:443: wsarecv: An existing connection was forcibly closed by the remote host.
ignoring error for \D\WindowsImageBackup\server\Backup 2023-11-01 071059\31845d83-74d5-4bce-a53e-a0356e5bf4f2.vhdx: StreamPack: ReadFull: read tcp 10.11.12.13:24818->1.2.3.4:443: wsarecv: An existing connection was forcibly closed by the remote host.
ignoring error for \D\WindowsImageBackup\server\Backup 2023-11-01 071059\0d14c486-c3a0-46ea-9f3e-1c07b67894f5_Writerd61d61c8-d73a-4eee-8cdd-f6f9786b7124.xml: StreamPack: ReadFull: read tcp 10.11.12.13:24816->1.2.3.4:443: wsarecv: An existing connection was forcibly closed by the remote host.
ignoring error for \D\WindowsImageBackup\server\Backup 2023-11-01 071059\31845d83-74d5-4bce-a53e-a0356e5bf4f2.vhdx: StreamPack: ReadFull: read tcp 10.11.12.13:37428->1.2.3.4:443: wsarecv: An existing connection was forcibly closed by the remote host.
ignoring error for \D\WindowsImageBackup\server\Backup 2023-11-01 071059\0d14c486-c3a0-46ea-9f3e-1c07b67894f5_Writere8132975-6f93-4464-a53e-1050253ae220.xml: StreamPack: ReadFull: read tcp 10.11.12.13:24816->1.2.3.4:443: wsarecv: An existing connection was forcibly closed by the remote host.
ignoring error for \D\WindowsImageBackup\server\Backup 2023-11-01 071059\0d14c486-c3a0-46ea-9f3e-1c07b67894f5_Writer0bada1de-01a9-4625-8278-69e735f39dd2.xml: StreamPack: ReadFull: read tcp 10.11.12.13:24816->1.2.3.4:443: wsarecv: An existing connection was forcibly closed by the remote host.
ignoring error for \D\WindowsImageBackup\server\Backup 2023-11-01 071059\0d14c486-c3a0-46ea-9f3e-1c07b67894f5_Writer1072ae1c-e5a7-4ea1-9e4a-6f7964656570.xml: StreamPack: ReadFull: read tcp 10.11.12.13:24816->1.2.3.4:443: wsarecv: An existing connection was forcibly closed by the remote host.
ignoring error for \D\WindowsImageBackup\server\Backup 2023-11-01 071059\0d14c486-c3a0-46ea-9f3e-1c07b67894f5_AdditionalFilesc3b9f3c7-5e52-4d5e-8b20-19adc95a34c7.xml: StreamPack: ReadFull: read tcp 10.11.12.13:24816->1.2.3.4:443: wsarecv: An existing connection was forcibly closed by the remote host.
ignoring error for \D\WindowsImageBackup\server\Backup 2023-11-01 071059\0d14c486-c3a0-46ea-9f3e-1c07b67894f5_Writer76fe1ac4-15f7-4bcd-987e-8e1acb462fb7.xml: StreamPack: ReadFull: read tcp 10.11.12.13:24816->1.2.3.4:443: wsarecv: An existing connection was forcibly closed by the remote host.
ignoring error for \D\WindowsImageBackup\server\Backup 2023-11-01 071059\0d14c486-c3a0-46ea-9f3e-1c07b67894f5_Writerafbab4a2-367d-4d15-a586-71dbb18f8485.xml: StreamPack: ReadFull: read tcp 10.11.12.13:24816->1.2.3.4:443: wsarecv: An existing connection was forcibly closed by the remote host.
ignoring error for \D\WindowsImageBackup\server\Logs\Backup_Error-01-11-2023_07-10-59.log: StreamPack: ReadFull: read tcp 10.11.12.13:24816->1.2.3.4:443: wsarecv: An existing connection was forcibly closed by the remote host.
ignoring error for \D\WindowsImageBackup\server\Backup 2023-11-01 071059\0d14c486-c3a0-46ea-9f3e-1c07b67894f5_Writer2a40fd15-dfca-4aa8-a654-1f8c654603f6.xml: StreamPack: ReadFull: read tcp 10.11.12.13:24816->1.2.3.4:443: wsarecv: An existing connection was forcibly closed by the remote host.
ignoring error for \D\WindowsImageBackup\server\Catalog\BackupGlobalCatalog: StreamPack: ReadFull: read tcp 10.11.12.13:24816->1.2.3.4:443: wsarecv: An existing connection was forcibly closed by the remote host.
ignoring error for \D\WindowsImageBackup\server\Backup 2023-11-01 071059\0d14c486-c3a0-46ea-9f3e-1c07b67894f5_Writer542da469-d3e1-473c-9f4f-7847f01fc64f.xml: StreamPack: ReadFull: read tcp 10.11.12.13:24816->1.2.3.4:443: wsarecv: An existing connection was forcibly closed by the remote host.
ignoring error for \D\WindowsImageBackup\server\Backup 2023-11-01 071059\0d14c486-c3a0-46ea-9f3e-1c07b67894f5_Writer59b1f0cf-90ef-465f-9609-6ca8b2938366.xml: StreamPack: ReadFull: read tcp 10.11.12.13:24816->1.2.3.4:443: wsarecv: An existing connection was forcibly closed by the remote host.
ignoring error for \D\WindowsImageBackup\server\Backup 2023-11-01 071059\0d14c486-c3a0-46ea-9f3e-1c07b67894f5_Writera6ad56c2-b509-4e6c-bb19-49d8f43532f0.xml: StreamPack: ReadFull: read tcp 10.11.12.13:24816->1.2.3.4:443: wsarecv: An existing connection was forcibly closed by the remote host.
ignoring error for \D\WindowsImageBackup\server\Backup 2023-11-01 071059\0d14c486-c3a0-46ea-9f3e-1c07b67894f5_Writer41e12264-35d8-479b-8e5c-9b23d1dad37e.xml: StreamPack: ReadFull: read tcp 10.11.12.13:24816->1.2.3.4:443: wsarecv: An existing connection was forcibly closed by the remote host.
ignoring error for \D\WindowsImageBackup\server\Backup 2023-11-01 071059\0d14c486-c3a0-46ea-9f3e-1c07b67894f5_Writer7e47b561-971a-46e6-96b9-696eeaa53b2a.xml: StreamPack: ReadFull: read tcp 10.11.12.13:24816->1.2.3.4:443: wsarecv: An existing connection was forcibly closed by the remote host.
ignoring error for \D\WindowsImageBackup\server\Backup 2023-11-01 071059\31845d83-74d5-4bce-a53e-a0356e5bf4f2.vhdx: StreamPack: ReadFull: read tcp 10.11.12.13:24816->1.2.3.4:443: wsarecv: An existing connection was forcibly closed by the remote host.
ignoring error for \D\WindowsImageBackup\server\Backup 2023-11-01 071059\BackupSpecs.xml: StreamPack: ReadFull: read tcp 10.11.12.13:24816->1.2.3.4:443: wsarecv: An existing connection was forcibly closed by the remote host.
ignoring error for \D\WindowsImageBackup\server\MediaId: StreamPack: ReadFull: read tcp 10.11.12.13:24816->1.2.3.4:443: wsarecv: An existing connection was forcibly closed by the remote host.
ignoring error for \D\WindowsImageBackup\server\Backup 2023-11-01 071059\0d14c486-c3a0-46ea-9f3e-1c07b67894f5_Components.xml: StreamPack: ReadFull: read tcp 10.11.12.13:24816->1.2.3.4:443: wsarecv: An existing connection was forcibly closed by the remote host.
ignoring error for \D\WindowsImageBackup\server\Backup 2023-11-01 071059\0d14c486-c3a0-46ea-9f3e-1c07b67894f5_Writer75dfb225-e2e4-4d39-9ac9-ffaff65ddf06.xml: StreamPack: ReadFull: read tcp 10.11.12.13:24816->1.2.3.4:443: wsarecv: An existing connection was forcibly closed by the remote host.
ignoring error for \D\WindowsImageBackup\server\Backup 2023-11-01 071059\0d14c486-c3a0-46ea-9f3e-1c07b67894f5_Writerbe000cbe-11fe-4426-9c58-531aa6355fc4.xml: StreamPack: ReadFull: read tcp 10.11.12.13:24816->1.2.3.4:443: wsarecv: An existing connection was forcibly closed by the remote host.
ignoring error for \D\WindowsImageBackup\server\Backup 2023-11-01 071059\0d14c486-c3a0-46ea-9f3e-1c07b67894f5_RegistryExcludes.xml: StreamPack: ReadFull: read tcp 10.11.12.13:24816->1.2.3.4:443: wsarecv: An existing connection was forcibly closed by the remote host.
ignoring error for \D\WindowsImageBackup\server\Backup 2023-11-01 071059\0d14c486-c3a0-46ea-9f3e-1c07b67894f5_Writer4969d978-be47-48b0-b100-f328f07ac1e0.xml: StreamPack: ReadFull: read tcp 10.11.12.13:24816->1.2.3.4:443: wsarecv: An existing connection was forcibly closed by the remote host.
ignoring error for \D\WindowsImageBackup\server\Backup 2023-11-01 071059\0d14c486-c3a0-46ea-9f3e-1c07b67894f5_Writer4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f.xml: StreamPack: ReadFull: read tcp 10.11.12.13:24816->1.2.3.4:443: wsarecv: An existing connection was forcibly closed by the remote host.
ignoring error for \D\WindowsImageBackup\server\Catalog\GlobalCatalog: StreamPack: ReadFull: read tcp 10.11.12.13:24816->1.2.3.4:443: wsarecv: An existing connection was forcibly closed by the remote host.
Load(<data/8337c383f6>, 5599356, 11202383) returned error, retrying after 502.312451ms: read tcp 10.11.12.13:10117->1.2.3.4:443: wsarecv: An existing connection was forcibly closed by the remote host.
Load(<data/8337c383f6>, 5599356, 11202383) operation successful after 1 retries
Load(<data/f153121586>, 17233225, 0) returned error, retrying after 644.634766ms: ReadFull: read tcp 10.11.12.13:17761->1.2.3.4:443: wsarecv: An existing connection was forcibly closed by the remote host.
Load(<data/b5d7bafb21>, 9956283, 6351043) returned error, retrying after 474.278926ms: read tcp 10.11.12.13:17769->1.2.3.4:443: wsarecv: An existing connection was forcibly closed by the remote host.
Load(<data/5380a41969>, 17823475, 0) returned error, retrying after 655.882519ms: ReadFull: read tcp 10.11.12.13:17775->1.2.3.4:443: wsarecv: An existing connection was forcibly closed by the remote host.
Load(<data/ac1478ab01>, 15550945, 0) returned error, retrying after 371.79632ms: read tcp 10.11.12.13:17780->1.2.3.4:443: wsarecv: An existing connection was forcibly closed by the remote host.
Load(<data/b5d7bafb21>, 9956283, 6351043) operation successful after 1 retries
Load(<data/ac1478ab01>, 15550945, 0) operation successful after 1 retries
Load(<data/f153121586>, 17233225, 0) operation successful after 1 retries
Load(<data/5380a41969>, 17823475, 0) operation successful after 1 retries
Summary: Restored 25 / 32 files/dirs (1.270 TiB / 1.271 TiB) in 3:33:50
Fatal: There were 29 errors

After restore I do hash check on restored file:

certutil -hashfile 31845d83-74d5-4bce-a53e-a0356e5bf4f2.vhdx

and hash differs from the source file I sent to backup.
I tried different combinations of parameters on backup and restore with no success:

  1. Using rclone to serve restic
  2. –use-fs-snapshot
  3. -o s3.list-objects-v1=true
  4. -o s3.region=“provider-specific”
  5. –pack-size 8
  6. –pack-size 32

I tried creating new S3 repo, still no success.
I might think someting wrong with my S3 provider, but when I tried Kopia it backed up and restored the file with full hash match.
Is there anything I can try else to figure out problem?

2

My Linux backup command line looks like this. It might be worth seeing what happens when you change the URL pattern. Your way may be completely valid, or it may have found an odd edge case that works for backup and not restore. I don’t think this will solve the problem, but it’s worth a shot because it’s quick and easy.

/opt/restic/restic backup --repo s3:s3.amazonaws.com/bucket-name

Next, double check that the repo is created in S3? i.e. the data / index / keys folders are created, and there’s data in the data folder.

It looks to me like S3 is refusing the connection. Suggest you first check CloudTrail to see if there’s anything interesting in the log, but there probably won’t be.

Next turn on or create a CloudTrail that records S3 data events. Do a backup, wait 10 minutes (only to make it easier to find logs), then try a restore. Have a look in CloudTrail for one data store event during backup then one data retrieve event during restore. I’d like to see the CloudTrail event, but that might tell you the answer without you posting it.

Please report back what works and what doesn’t work, with as much information and log data as you’re comfortable sharing about what happens when you try the suggestions above.