Thank you Michael,
I built 0.12.0 with debugging The bucket permissions are such that the user can read/write the bucket but can’t enumerate the buckets, and the debug output looks like restic is failing on checking whether the bucket exists or not.
The bucket does exist and is reachable with both s3cmd and rclone - though the rclone configuration took quite a bit of puzzling to get working. For rclone, the critical bit was the region = other-v2-signature
.
The root cause seems to be restic is using v4 signatures and the provider I’m using being on a swift version that only does v2 signatures at the moment (they’re upgrading but it’s not released yet). v2 signature are deprecated on Amazon S3 but still supported by rclone.
The whole rclone config block is:
[cloud]
type = s3
provider = Ceph
env_auth = true
access_key_id = [REDACTED]
secret_access_key = [REDACTED]
region = other-v2-signature
endpoint = https://object-storage.nz-por-1.catalystcloud.io
acl = private
The debug log from restic is as follows:
2021/05/23 22:21:31 restic/main.go:95 main.main 1 main []string{"./restic", "ls"}
2021/05/23 22:21:31 restic/main.go:96 main.main 1 restic 0.12.0 (v0.12.0-0-g27f24133) compiled with go1.13.8 on linux/amd64
2021/05/23 22:21:31 restic/lock.go:119 main.unlockAll 1 unlocking 0 locks
2021/05/23 22:21:37 restic/main.go:95 main.main 1 main []string{"./restic", "init"}
2021/05/23 22:21:37 restic/main.go:96 main.main 1 restic 0.12.0 (v0.12.0-0-g27f24133) compiled with go1.13.8 on linux/amd64
2021/05/23 22:21:37 restic/global.go:751 main.create 1 parsing location s3:https://object-storage.nz-por-1.catalystcloud.io/bucket-name
2021/05/23 22:21:37 restic/global.go:574 main.parseConfig 1 opening s3 repository at s3.Config{Endpoint:"object-storage.nz-por-1.catalystcloud.io", UseHTTP:false, KeyID:"[REDACTED]", Secret:"[REDACTED]", Bucket:"bucket-name", Prefix:"", Layout:"", StorageClass:"", Connections:0x5, MaxRetries:0x0, Region:"", BucketLookup:"", ListObjectsV1:false}
2021/05/23 22:21:37 s3/s3.go:38 s3.open 1 open, config s3.Config{Endpoint:"object-storage.nz-por-1.catalystcloud.io", UseHTTP:false, KeyID:"[REDACTED]", Secret:"[REDACTED]", Bucket:"bucket-name", Prefix:"", Layout:"", StorageClass:"", Connections:0x5, MaxRetries:0x0, Region:"", BucketLookup:"", ListObjectsV1:false}
2021/05/23 22:21:37 backend/layout.go:139 backend.ParseLayout 1 parse layout string "" for backend at
2021/05/23 22:21:37 backend/layout.go:99 backend.DetectLayout 1 detect layout at
2021/05/23 22:21:37 s3/s3.go:198 s3.(*Backend).ReadDir 1 ReadDir(keys)
2021/05/23 22:21:37 s3/s3.go:208 s3.(*Backend).ReadDir 1 using ListObjectsV1(false)
2021/05/23 22:21:37 debug/round_tripper_debug.go:82 debug.loggingRoundTripper.RoundTrip 21 ------------ HTTP REQUEST -----------
GET /bucket-name/?location= HTTP/1.1
Host: object-storage.nz-por-1.catalystcloud.io
User-Agent: MinIO (linux; amd64) minio-go/v7.0.5
Authorization: AWS4-HMAC-SHA256 Credential=[REDACTED]/20210523/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=0ca96f166c706d434349ae112af84eae419c681a8a8a55c3f06d2c98d187f47a
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20210523T102137Z
Accept-Encoding: gzip
2021/05/23 22:21:38 debug/round_tripper_debug.go:95 debug.loggingRoundTripper.RoundTrip 21 ------------ HTTP RESPONSE ----------
HTTP/1.1 412 Precondition Failed
Content-Length: 7
Content-Type: text/html; charset=UTF-8
Date: Sun, 23 May 2021 10:21:38 GMT
Server: nginx/1.16.0
X-Trans-Id: tx538fcdc6682c44fbbaef0-0060aa2cb2
2021/05/23 22:21:38 s3/s3.go:198 s3.(*Backend).ReadDir 1 ReadDir(key)
2021/05/23 22:21:38 s3/s3.go:208 s3.(*Backend).ReadDir 1 using ListObjectsV1(false)
2021/05/23 22:21:38 debug/round_tripper_debug.go:82 debug.loggingRoundTripper.RoundTrip 48 ------------ HTTP REQUEST -----------
GET /bucket-name/?location= HTTP/1.1
Host: object-storage.nz-por-1.catalystcloud.io
User-Agent: MinIO (linux; amd64) minio-go/v7.0.5
Authorization: AWS4-HMAC-SHA256 Credential=[REDACTED]/20210523/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=0040dbbef13786beabaec63cf1da7b296ad6688b090a750845968dbfbec6b49f
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20210523T102138Z
Accept-Encoding: gzip
2021/05/23 22:21:38 debug/round_tripper_debug.go:95 debug.loggingRoundTripper.RoundTrip 48 ------------ HTTP RESPONSE ----------
HTTP/1.1 412 Precondition Failed
Content-Length: 7
Content-Type: text/html; charset=UTF-8
Date: Sun, 23 May 2021 10:21:38 GMT
Server: nginx/1.16.0
X-Trans-Id: tx96b4d6eac4ce45d9b8bcb-0060aa2cb2
2021/05/23 22:21:38 backend/layout.go:132 backend.DetectLayout 1 layout detection failed
2021/05/23 22:21:38 backend/layout.go:156 backend.ParseLayout 1 error: auto-detecting the filesystem layout failed, use default layout default
2021/05/23 22:21:38 backend/layout.go:139 backend.ParseLayout 1 parse layout string "default" for backend at
2021/05/23 22:21:38 debug/round_tripper_debug.go:82 debug.loggingRoundTripper.RoundTrip 1 ------------ HTTP REQUEST -----------
GET /bucket-name/?location= HTTP/1.1
Host: object-storage.nz-por-1.catalystcloud.io
User-Agent: MinIO (linux; amd64) minio-go/v7.0.5
Authorization: AWS4-HMAC-SHA256 Credential=[REDACTED]/20210523/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=0040dbbef13786beabaec63cf1da7b296ad6688b090a750845968dbfbec6b49f
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20210523T102138Z
Accept-Encoding: gzip
2021/05/23 22:21:38 debug/round_tripper_debug.go:95 debug.loggingRoundTripper.RoundTrip 1 ------------ HTTP RESPONSE ----------
HTTP/1.1 412 Precondition Failed
Content-Length: 7
Content-Type: text/html; charset=UTF-8
Date: Sun, 23 May 2021 10:21:38 GMT
Server: nginx/1.16.0
X-Trans-Id: tx724a1bdc1cd84051bcbde-0060aa2cb2
2021/05/23 22:21:38 s3/s3.go:153 s3.(*Backend).IsAccessDenied 1 IsAccessDenied(minio.ErrorResponse, minio.ErrorResponse{XMLName:xml.Name{Space:"", Local:""}, Code:"PreconditionFailed", Message:"At least one of the pre-conditions you specified did not hold", BucketName:"bucket-name", Key:"", RequestID:"", HostID:"", Region:"", Server:"", StatusCode:412})
2021/05/23 22:21:38 s3/s3.go:136 s3.Create 1 BucketExists(bucket-name) returned err At least one of the pre-conditions you specified did not hold
2021/05/23 22:21:38 restic/lock.go:119 main.unlockAll 1 unlocking 0 locks
2021/05/23 22:27:10 restic/main.go:95 main.main 1 main []string{"./restic", "init"}
2021/05/23 22:27:10 restic/main.go:96 main.main 1 restic 0.12.0 (v0.12.0-0-g27f24133) compiled with go1.13.8 on linux/amd64
2021/05/23 22:27:10 restic/global.go:751 main.create 1 parsing location s3:https://object-storage.nz-por-1.catalystcloud.io/bucket-name
2021/05/23 22:27:10 restic/global.go:574 main.parseConfig 1 opening s3 repository at s3.Config{Endpoint:"object-storage.nz-por-1.catalystcloud.io", UseHTTP:false, KeyID:"[REDACTED]", Secret:"[REDACTED]", Bucket:"bucket-name", Prefix:"", Layout:"", StorageClass:"", Connections:0x5, MaxRetries:0x0, Region:"", BucketLookup:"", ListObjectsV1:false}
2021/05/23 22:27:10 s3/s3.go:38 s3.open 1 open, config s3.Config{Endpoint:"object-storage.nz-por-1.catalystcloud.io", UseHTTP:false, KeyID:"[REDACTED]", Secret:"[REDACTED]", Bucket:"bucket-name", Prefix:"", Layout:"", StorageClass:"", Connections:0x5, MaxRetries:0x0, Region:"", BucketLookup:"", ListObjectsV1:false}
2021/05/23 22:27:10 backend/layout.go:139 backend.ParseLayout 1 parse layout string "" for backend at
2021/05/23 22:27:10 backend/layout.go:99 backend.DetectLayout 1 detect layout at
2021/05/23 22:27:10 s3/s3.go:198 s3.(*Backend).ReadDir 1 ReadDir(keys)
2021/05/23 22:27:10 s3/s3.go:208 s3.(*Backend).ReadDir 1 using ListObjectsV1(false)
2021/05/23 22:27:10 debug/round_tripper_debug.go:82 debug.loggingRoundTripper.RoundTrip 19 ------------ HTTP REQUEST -----------
GET /bucket-name/?location= HTTP/1.1
Host: object-storage.nz-por-1.catalystcloud.io
User-Agent: MinIO (linux; amd64) minio-go/v7.0.5
Authorization: AWS4-HMAC-SHA256 Credential=[REDACTED]/20210523/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=195f2a095fa373492b41c104e8164a6b7b8fec3d027a0b544e924aa04890d624
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20210523T102710Z
Accept-Encoding: gzip
2021/05/23 22:27:11 debug/round_tripper_debug.go:95 debug.loggingRoundTripper.RoundTrip 19 ------------ HTTP RESPONSE ----------
HTTP/1.1 412 Precondition Failed
Content-Length: 7
Content-Type: text/html; charset=UTF-8
Date: Sun, 23 May 2021 10:27:11 GMT
Server: nginx/1.16.0
X-Trans-Id: txe0505be44b624454be446-0060aa2dff
2021/05/23 22:27:11 s3/s3.go:198 s3.(*Backend).ReadDir 1 ReadDir(key)
2021/05/23 22:27:11 s3/s3.go:208 s3.(*Backend).ReadDir 1 using ListObjectsV1(false)
2021/05/23 22:27:11 debug/round_tripper_debug.go:82 debug.loggingRoundTripper.RoundTrip 38 ------------ HTTP REQUEST -----------
GET /bucket-name/?location= HTTP/1.1
Host: object-storage.nz-por-1.catalystcloud.io
User-Agent: MinIO (linux; amd64) minio-go/v7.0.5
Authorization: AWS4-HMAC-SHA256 Credential=[REDACTED]/20210523/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=9f526a92c25087d5b0ecfb9c6095c2988a3384564068883d4c01ceb7d6ff1fb0
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20210523T102711Z
Accept-Encoding: gzip
2021/05/23 22:27:11 debug/round_tripper_debug.go:95 debug.loggingRoundTripper.RoundTrip 38 ------------ HTTP RESPONSE ----------
HTTP/1.1 412 Precondition Failed
Content-Length: 7
Content-Type: text/html; charset=UTF-8
Date: Sun, 23 May 2021 10:27:11 GMT
Server: nginx/1.16.0
X-Trans-Id: txf28c0c958bd94268b698a-0060aa2dff
2021/05/23 22:27:11 backend/layout.go:132 backend.DetectLayout 1 layout detection failed
2021/05/23 22:27:11 backend/layout.go:156 backend.ParseLayout 1 error: auto-detecting the filesystem layout failed, use default layout default
2021/05/23 22:27:11 backend/layout.go:139 backend.ParseLayout 1 parse layout string "default" for backend at
2021/05/23 22:27:11 debug/round_tripper_debug.go:82 debug.loggingRoundTripper.RoundTrip 1 ------------ HTTP REQUEST -----------
GET /bucket-name/?location= HTTP/1.1
Host: object-storage.nz-por-1.catalystcloud.io
User-Agent: MinIO (linux; amd64) minio-go/v7.0.5
Authorization: AWS4-HMAC-SHA256 Credential=[REDACTED]/20210523/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=9f526a92c25087d5b0ecfb9c6095c2988a3384564068883d4c01ceb7d6ff1fb0
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20210523T102711Z
Accept-Encoding: gzip
2021/05/23 22:27:11 debug/round_tripper_debug.go:95 debug.loggingRoundTripper.RoundTrip 1 ------------ HTTP RESPONSE ----------
HTTP/1.1 412 Precondition Failed
Content-Length: 7
Content-Type: text/html; charset=UTF-8
Date: Sun, 23 May 2021 10:27:11 GMT
Server: nginx/1.16.0
X-Trans-Id: tx5abb15255c674343b2751-0060aa2dff
2021/05/23 22:27:11 s3/s3.go:153 s3.(*Backend).IsAccessDenied 1 IsAccessDenied(minio.ErrorResponse, minio.ErrorResponse{XMLName:xml.Name{Space:"", Local:""}, Code:"PreconditionFailed", Message:"At least one of the pre-conditions you specified did not hold", BucketName:"bucket-name", Key:"", RequestID:"", HostID:"", Region:"", Server:"", StatusCode:412})
2021/05/23 22:27:11 s3/s3.go:136 s3.Create 1 BucketExists(bucket-name) returned err At least one of the pre-conditions you specified did not hold
2021/05/23 22:27:11 restic/lock.go:119 main.unlockAll 1 unlocking 0 locks