Restic + Windows + AWS S3 does not work

Getting Help
1

Hi. I’m trying to set up restic backup on windows to Amazon S3 following this instruction
https://restic.readthedocs.io/en/latest/080_examples.html

I’ve created aws user with static key and permissions to s3 bucket and bucket at eu-central-1 region

C:\Users\andreyche>restic version
restic 0.16.4 compiled with go1.21.6 on windows/amd64

I set enviroment vars:

set AWS_DEFAULT_REGION="eu-central-1"
set RESTIC_REPOSITORY=s3:https://s3.amazonaws.com/my-bucket-name
set AWS_ACCESS_KEY_ID="XXX"
set AWS_SECRET_ACCESS_KEY="XXX"
set RESTIC_PASSWORD="XXX"

Now trying to init repo and get error:

C:\Users\andreyche>restic init
Fatal: create repository at s3:https://s3.amazonaws.com/my-bucket-name failed: client.BucketExists: 301 Moved Permanently

I turned debug log and tried again:

set DEBUG_LOG=D:\temp\restic.txt
restic init

Here is debug log:

2024/03/25 11:41:56 restic/main.go:106	main.main	1	main []string{"restic", "init"}
2024/03/25 11:41:56 restic/main.go:107	main.main	1	restic 0.16.4 compiled with go1.21.6 on windows/amd64
2024/03/25 11:41:56 restic/global.go:633	main.create	1	parsing location s3:https://s3.amazonaws.com/my-bucket-name
2024/03/25 11:41:56 restic/global.go:569	main.parseConfig	1	opening s3 repository at &s3.Config{Endpoint:"s3.amazonaws.com", UseHTTP:false, KeyID:"\"XXX\"", Secret:"**redacted**", Bucket:"my-bucket-name", Prefix:"", Layout:"", StorageClass:"", Connections:0x5, MaxRetries:0x0, Region:"\"eu-central-1\"", BucketLookup:"", ListObjectsV1:false}
2024/03/25 11:41:56 s3/s3.go:42	s3.open	1	open, config s3.Config{Endpoint:"s3.amazonaws.com", UseHTTP:false, KeyID:"\"XXX\"", Secret:"**redacted**", Bucket:"my-bucket-name", Prefix:"", Layout:"", StorageClass:"", Connections:0x5, MaxRetries:0x0, Region:"\"eu-central-1\"", BucketLookup:"", ListObjectsV1:false}
2024/03/25 11:41:56 layout/layout.go:139	layout.ParseLayout	1	parse layout string "" for backend at 
2024/03/25 11:41:56 layout/layout.go:99	layout.DetectLayout	1	detect layout at 
2024/03/25 11:41:56 s3/s3.go:200	s3.(*Backend).ReadDir	1	ReadDir(keys)
2024/03/25 11:41:56 s3/s3.go:210	s3.(*Backend).ReadDir	1	using ListObjectsV1(false)
2024/03/25 11:41:56 debug/round_tripper.go:93	debug.loggingRoundTripper.RoundTrip	26	------------  HTTP REQUEST -----------
GET /?delimiter=%2F&encoding-type=url&fetch-owner=true&list-type=2&prefix=keys%2F HTTP/1.1
Host: my-bucket-name.s3.dualstack.us-east-1.amazonaws.com
User-Agent: MinIO (windows; amd64) minio-go/v7.0.66
Authorization: **redacted**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240325T084156Z
Accept-Encoding: gzip


2024/03/25 11:41:56 debug/round_tripper.go:110	debug.loggingRoundTripper.RoundTrip	26	------------  HTTP RESPONSE ----------
HTTP/1.1 301 Moved Permanently
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Mon, 25 Mar 2024 08:41:47 GMT
Server: AmazonS3
X-Amz-Bucket-Region: eu-central-1
X-Amz-Id-2: rNIfOmohA74c1/pG4ysk4R8hHOSgOGuGwVtsC+X/GzBKbmlihMUhQPqjmRhHOK/0qigXxP6CG3Q=
X-Amz-Request-Id: QRCVNCXQTWMFM754


2024/03/25 11:41:56 s3/s3.go:200	s3.(*Backend).ReadDir	1	ReadDir(key)
2024/03/25 11:41:56 s3/s3.go:210	s3.(*Backend).ReadDir	1	using ListObjectsV1(false)
2024/03/25 11:41:56 debug/round_tripper.go:93	debug.loggingRoundTripper.RoundTrip	30	------------  HTTP REQUEST -----------
GET /?delimiter=%2F&encoding-type=url&fetch-owner=true&list-type=2&prefix=key%2F HTTP/1.1
Host: my-bucket-name.s3.dualstack.us-east-1.amazonaws.com
User-Agent: MinIO (windows; amd64) minio-go/v7.0.66
Authorization: **redacted**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240325T084156Z
Accept-Encoding: gzip

....

2024/03/25 11:41:57 s3/s3.go:166	s3.isAccessDenied	1	isAccessDenied(minio.ErrorResponse, minio.ErrorResponse{XMLName:xml.Name{Space:"", Local:""}, Code:"301 Moved Permanently", Message:"301 Moved Permanently", BucketName:"my-bucket-name", Key:"", Resource:"", RequestID:"QRCYE1PE9E88RK70", HostID:"tNLswafgtQckevExuZgvs4cijU1FMAMVR742Yyix8MJSMxCHEOdinbFj8WMsaEjRHAAXpIwf+2I=", Region:"eu-central-1", Server:"", StatusCode:301})
2024/03/25 11:41:57 s3/s3.go:149	s3.Create	1	BucketExists(my-bucket-name) returned err 301 Moved Permanently
2024/03/25 11:41:57 restic/cleanup.go:87	main.Exit	1	exiting with status code 1

Looks like restic ignores env var AWS_DEFAULT_REGION and tries to connect to us-east-1 region

I tried to set region in command line args and get Access Denied:

C:\Users\andreyche>restic -o s3.region="eu-central-1" init
Fatal: create key in repository at s3:https://s3.amazonaws.com/my-bucket-name failed: Stat: Access Denied.

Here debug logs:

2024/03/25 11:43:23 restic/main.go:106	main.main	1	main []string{"restic", "-o", "s3.region=eu-central-1", "init"}
2024/03/25 11:43:23 restic/main.go:107	main.main	1	restic 0.16.4 compiled with go1.21.6 on windows/amd64
2024/03/25 11:43:23 restic/global.go:633	main.create	1	parsing location s3:https://s3.amazonaws.com/my-bucket-name
2024/03/25 11:43:23 restic/global.go:569	main.parseConfig	1	opening s3 repository at &s3.Config{Endpoint:"s3.amazonaws.com", UseHTTP:false, KeyID:"\"XXX\"", Secret:"**redacted**", Bucket:"my-bucket-name", Prefix:"", Layout:"", StorageClass:"", Connections:0x5, MaxRetries:0x0, Region:"eu-central-1", BucketLookup:"", ListObjectsV1:false}
2024/03/25 11:43:23 s3/s3.go:42	s3.open	1	open, config s3.Config{Endpoint:"s3.amazonaws.com", UseHTTP:false, KeyID:"\"XXX\"", Secret:"**redacted**", Bucket:"my-bucket-name", Prefix:"", Layout:"", StorageClass:"", Connections:0x5, MaxRetries:0x0, Region:"eu-central-1", BucketLookup:"", ListObjectsV1:false}
2024/03/25 11:43:23 layout/layout.go:139	layout.ParseLayout	1	parse layout string "" for backend at 
2024/03/25 11:43:23 layout/layout.go:99	layout.DetectLayout	1	detect layout at 
2024/03/25 11:43:23 s3/s3.go:200	s3.(*Backend).ReadDir	1	ReadDir(keys)
2024/03/25 11:43:23 s3/s3.go:210	s3.(*Backend).ReadDir	1	using ListObjectsV1(false)
2024/03/25 11:43:23 debug/round_tripper.go:93	debug.loggingRoundTripper.RoundTrip	50	------------  HTTP REQUEST -----------
GET /?delimiter=%2F&encoding-type=url&fetch-owner=true&list-type=2&prefix=keys%2F HTTP/1.1
Host: my-bucket-name.s3.dualstack.eu-central-1.amazonaws.com
User-Agent: MinIO (windows; amd64) minio-go/v7.0.66
Authorization: **redacted**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240325T084323Z
Accept-Encoding: gzip


2024/03/25 11:43:23 debug/round_tripper.go:110	debug.loggingRoundTripper.RoundTrip	50	------------  HTTP RESPONSE ----------
HTTP/1.1 403 Forbidden
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Mon, 25 Mar 2024 08:43:14 GMT
Server: AmazonS3
X-Amz-Bucket-Region: eu-central-1
X-Amz-Id-2: AmeKxYEB/Mar47ovFzVk++UYiLzuqMZEKnO+vr5IvdxCIgZtoYdBd91oFmGBc+DeFLa9nk+IBCU=

...

2024/03/25 11:43:24 logger/log.go:54	logger.(*Backend).Stat	1	  stat err Stat: Access Denied.
2024/03/25 11:43:24 logger/log.go:24	logger.(*Backend).IsNotExist	1	IsNotExist(*errors.withStack, Stat: Access Denied., false)
2024/03/25 11:43:24 restic/cleanup.go:87	main.Exit	1	exiting with status code 1

Why I get Access Denied? The thing is on linux with total the same parameters it works fine:

# restic version
restic 0.12.1 compiled with go1.18.1 on linux/amd64

export AWS_DEFAULT_REGION="eu-central-1"
export RESTIC_REPOSITORY="s3:https://s3.amazonaws.com/my-bucket-name"
export AWS_ACCESS_KEY_ID="XXX"
export AWS_SECRET_ACCESS_KEY="XXX"
export RESTIC_PASSWORD="XXX"

# restic init
created restic repository d3e4e43f9d at s3:https://s3.amazonaws.com/my-bucket-name

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.
2

try with:

set RESTIC_REPOSITORY="s3:https://s3.amazonaws.com/my-bucket-name"
3

Actually firstly I tried it:
set RESTIC_REPOSITORY="s3:https://s3.amazonaws.com/my-bucket-name"

and get “invalid backend”:

C:\Users\andreyche>restic init
Fatal: create repository at "s3:https://s3.amazonaws.com/my-bucket-name" failed: invalid backend
If the repository is in a local directory, you need to add a `local:` prefix

On windows quotes does not works

4

I am not Windows person but I do expect that problem is with env variables values.

If quotes do not work then why to use them at all for other variables?

1 Like
5

Thank you! I’ve removed quotes in all vars and it started to work :grimacing: :man_facepalming:

1 Like
6

Glad it works. Windows and quoting or globbing is always full of surprises for me:)

7

This also explains why restic “ignored” env AWS_DEFAULT_REGION, because it was containing quotes in region name so restic could not found such region and used default us-east-1 :grimacing:

1 Like