Openstack S3 Error

amuckart · May 21, 2021, 6:05am

Hi all,

I’m using restic 0.12.0 compiled with go1.15.8 on linux/amd64 and trying to init a repo in an OpenStack object store using the S3 protocol.

For various reasons I can’t use swift directly for this (it’s testing) but when I try and init the repo I get the following error:

$ restic -vvv init
Fatal: create repository at s3:https://object-storage.nz-por-1.catalystcloud.io/test-server-backups failed: client.BucketExists: At least one of the pre-conditions you specified did not hold

I get this regardless of whether the bucket exists or not.

Using the same credentials from the same server I can connect with s3cmd but both restic and rclone give me this error regardless of the access permissions of the keys I’m using. I have a test user with read/write access to the single bucket, and another use with full admin access on the openstack project and they behave the same.

Any pointers would be greatly appreciated.

Thanks.

MichaelEischer · May 22, 2021, 8:46pm

The best way to investigate this is probably to use a debug build of restic as described here: restic/CONTRIBUTING.md at master · restic/restic · GitHub . Feel free to ask if you need a more detailed instruction. The debug log should provide more information on what went wrong, including the full reply from the server.

restic initially checks that the bucked exists and expects S3 to either reply with details about the bucket, permission denied or not such bucket.

amuckart · May 23, 2021, 11:25pm

Thank you Michael,

I built 0.12.0 with debugging The bucket permissions are such that the user can read/write the bucket but can’t enumerate the buckets, and the debug output looks like restic is failing on checking whether the bucket exists or not.

The bucket does exist and is reachable with both s3cmd and rclone - though the rclone configuration took quite a bit of puzzling to get working. For rclone, the critical bit was the region = other-v2-signature.

The root cause seems to be restic is using v4 signatures and the provider I’m using being on a swift version that only does v2 signatures at the moment (they’re upgrading but it’s not released yet). v2 signature are deprecated on Amazon S3 but still supported by rclone.

The whole rclone config block is:

[cloud]
type = s3
provider = Ceph
env_auth = true
access_key_id = [REDACTED]
secret_access_key = [REDACTED]
region = other-v2-signature
endpoint = https://object-storage.nz-por-1.catalystcloud.io
acl = private

The debug log from restic is as follows:

2021/05/23 22:21:31 restic/main.go:95   main.main       1       main []string{"./restic", "ls"}
2021/05/23 22:21:31 restic/main.go:96   main.main       1       restic 0.12.0 (v0.12.0-0-g27f24133) compiled with go1.13.8 on linux/amd64
2021/05/23 22:21:31 restic/lock.go:119  main.unlockAll  1       unlocking 0 locks
2021/05/23 22:21:37 restic/main.go:95   main.main       1       main []string{"./restic", "init"}
2021/05/23 22:21:37 restic/main.go:96   main.main       1       restic 0.12.0 (v0.12.0-0-g27f24133) compiled with go1.13.8 on linux/amd64
2021/05/23 22:21:37 restic/global.go:751        main.create     1       parsing location s3:https://object-storage.nz-por-1.catalystcloud.io/bucket-name
2021/05/23 22:21:37 restic/global.go:574        main.parseConfig        1       opening s3 repository at s3.Config{Endpoint:"object-storage.nz-por-1.catalystcloud.io", UseHTTP:false, KeyID:"[REDACTED]", Secret:"[REDACTED]", Bucket:"bucket-name", Prefix:"", Layout:"", StorageClass:"", Connections:0x5, MaxRetries:0x0, Region:"", BucketLookup:"", ListObjectsV1:false}
2021/05/23 22:21:37 s3/s3.go:38 s3.open 1       open, config s3.Config{Endpoint:"object-storage.nz-por-1.catalystcloud.io", UseHTTP:false, KeyID:"[REDACTED]", Secret:"[REDACTED]", Bucket:"bucket-name", Prefix:"", Layout:"", StorageClass:"", Connections:0x5, MaxRetries:0x0, Region:"", BucketLookup:"", ListObjectsV1:false}
2021/05/23 22:21:37 backend/layout.go:139       backend.ParseLayout     1       parse layout string "" for backend at 
2021/05/23 22:21:37 backend/layout.go:99        backend.DetectLayout    1       detect layout at 
2021/05/23 22:21:37 s3/s3.go:198        s3.(*Backend).ReadDir   1       ReadDir(keys)
2021/05/23 22:21:37 s3/s3.go:208        s3.(*Backend).ReadDir   1       using ListObjectsV1(false)
2021/05/23 22:21:37 debug/round_tripper_debug.go:82     debug.loggingRoundTripper.RoundTrip     21      ------------  HTTP REQUEST -----------
GET /bucket-name/?location= HTTP/1.1
Host: object-storage.nz-por-1.catalystcloud.io
User-Agent: MinIO (linux; amd64) minio-go/v7.0.5
Authorization: AWS4-HMAC-SHA256 Credential=[REDACTED]/20210523/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=0ca96f166c706d434349ae112af84eae419c681a8a8a55c3f06d2c98d187f47a
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20210523T102137Z
Accept-Encoding: gzip


2021/05/23 22:21:38 debug/round_tripper_debug.go:95     debug.loggingRoundTripper.RoundTrip     21      ------------  HTTP RESPONSE ----------
HTTP/1.1 412 Precondition Failed
Content-Length: 7
Content-Type: text/html; charset=UTF-8
Date: Sun, 23 May 2021 10:21:38 GMT
Server: nginx/1.16.0
X-Trans-Id: tx538fcdc6682c44fbbaef0-0060aa2cb2


2021/05/23 22:21:38 s3/s3.go:198        s3.(*Backend).ReadDir   1       ReadDir(key)
2021/05/23 22:21:38 s3/s3.go:208        s3.(*Backend).ReadDir   1       using ListObjectsV1(false)
2021/05/23 22:21:38 debug/round_tripper_debug.go:82     debug.loggingRoundTripper.RoundTrip     48      ------------  HTTP REQUEST -----------
GET /bucket-name/?location= HTTP/1.1
Host: object-storage.nz-por-1.catalystcloud.io
User-Agent: MinIO (linux; amd64) minio-go/v7.0.5
Authorization: AWS4-HMAC-SHA256 Credential=[REDACTED]/20210523/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=0040dbbef13786beabaec63cf1da7b296ad6688b090a750845968dbfbec6b49f
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20210523T102138Z
Accept-Encoding: gzip


2021/05/23 22:21:38 debug/round_tripper_debug.go:95     debug.loggingRoundTripper.RoundTrip     48      ------------  HTTP RESPONSE ----------
HTTP/1.1 412 Precondition Failed
Content-Length: 7
Content-Type: text/html; charset=UTF-8
Date: Sun, 23 May 2021 10:21:38 GMT
Server: nginx/1.16.0
X-Trans-Id: tx96b4d6eac4ce45d9b8bcb-0060aa2cb2


2021/05/23 22:21:38 backend/layout.go:132       backend.DetectLayout    1       layout detection failed
2021/05/23 22:21:38 backend/layout.go:156       backend.ParseLayout     1       error: auto-detecting the filesystem layout failed, use default layout default
2021/05/23 22:21:38 backend/layout.go:139       backend.ParseLayout     1       parse layout string "default" for backend at 
2021/05/23 22:21:38 debug/round_tripper_debug.go:82     debug.loggingRoundTripper.RoundTrip     1       ------------  HTTP REQUEST -----------
GET /bucket-name/?location= HTTP/1.1
Host: object-storage.nz-por-1.catalystcloud.io
User-Agent: MinIO (linux; amd64) minio-go/v7.0.5
Authorization: AWS4-HMAC-SHA256 Credential=[REDACTED]/20210523/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=0040dbbef13786beabaec63cf1da7b296ad6688b090a750845968dbfbec6b49f
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20210523T102138Z
Accept-Encoding: gzip


2021/05/23 22:21:38 debug/round_tripper_debug.go:95     debug.loggingRoundTripper.RoundTrip     1       ------------  HTTP RESPONSE ----------
HTTP/1.1 412 Precondition Failed
Content-Length: 7
Content-Type: text/html; charset=UTF-8
Date: Sun, 23 May 2021 10:21:38 GMT
Server: nginx/1.16.0
X-Trans-Id: tx724a1bdc1cd84051bcbde-0060aa2cb2


2021/05/23 22:21:38 s3/s3.go:153        s3.(*Backend).IsAccessDenied    1       IsAccessDenied(minio.ErrorResponse, minio.ErrorResponse{XMLName:xml.Name{Space:"", Local:""}, Code:"PreconditionFailed", Message:"At least one of the pre-conditions you specified did not hold", BucketName:"bucket-name", Key:"", RequestID:"", HostID:"", Region:"", Server:"", StatusCode:412})
2021/05/23 22:21:38 s3/s3.go:136        s3.Create       1       BucketExists(bucket-name) returned err At least one of the pre-conditions you specified did not hold
2021/05/23 22:21:38 restic/lock.go:119  main.unlockAll  1       unlocking 0 locks
2021/05/23 22:27:10 restic/main.go:95   main.main       1       main []string{"./restic", "init"}
2021/05/23 22:27:10 restic/main.go:96   main.main       1       restic 0.12.0 (v0.12.0-0-g27f24133) compiled with go1.13.8 on linux/amd64
2021/05/23 22:27:10 restic/global.go:751        main.create     1       parsing location s3:https://object-storage.nz-por-1.catalystcloud.io/bucket-name
2021/05/23 22:27:10 restic/global.go:574        main.parseConfig        1       opening s3 repository at s3.Config{Endpoint:"object-storage.nz-por-1.catalystcloud.io", UseHTTP:false, KeyID:"[REDACTED]", Secret:"[REDACTED]", Bucket:"bucket-name", Prefix:"", Layout:"", StorageClass:"", Connections:0x5, MaxRetries:0x0, Region:"", BucketLookup:"", ListObjectsV1:false}
2021/05/23 22:27:10 s3/s3.go:38 s3.open 1       open, config s3.Config{Endpoint:"object-storage.nz-por-1.catalystcloud.io", UseHTTP:false, KeyID:"[REDACTED]", Secret:"[REDACTED]", Bucket:"bucket-name", Prefix:"", Layout:"", StorageClass:"", Connections:0x5, MaxRetries:0x0, Region:"", BucketLookup:"", ListObjectsV1:false}
2021/05/23 22:27:10 backend/layout.go:139       backend.ParseLayout     1       parse layout string "" for backend at 
2021/05/23 22:27:10 backend/layout.go:99        backend.DetectLayout    1       detect layout at 
2021/05/23 22:27:10 s3/s3.go:198        s3.(*Backend).ReadDir   1       ReadDir(keys)
2021/05/23 22:27:10 s3/s3.go:208        s3.(*Backend).ReadDir   1       using ListObjectsV1(false)
2021/05/23 22:27:10 debug/round_tripper_debug.go:82     debug.loggingRoundTripper.RoundTrip     19      ------------  HTTP REQUEST -----------
GET /bucket-name/?location= HTTP/1.1
Host: object-storage.nz-por-1.catalystcloud.io
User-Agent: MinIO (linux; amd64) minio-go/v7.0.5
Authorization: AWS4-HMAC-SHA256 Credential=[REDACTED]/20210523/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=195f2a095fa373492b41c104e8164a6b7b8fec3d027a0b544e924aa04890d624
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20210523T102710Z
Accept-Encoding: gzip


2021/05/23 22:27:11 debug/round_tripper_debug.go:95     debug.loggingRoundTripper.RoundTrip     19      ------------  HTTP RESPONSE ----------
HTTP/1.1 412 Precondition Failed
Content-Length: 7
Content-Type: text/html; charset=UTF-8
Date: Sun, 23 May 2021 10:27:11 GMT
Server: nginx/1.16.0
X-Trans-Id: txe0505be44b624454be446-0060aa2dff


2021/05/23 22:27:11 s3/s3.go:198        s3.(*Backend).ReadDir   1       ReadDir(key)
2021/05/23 22:27:11 s3/s3.go:208        s3.(*Backend).ReadDir   1       using ListObjectsV1(false)
2021/05/23 22:27:11 debug/round_tripper_debug.go:82     debug.loggingRoundTripper.RoundTrip     38      ------------  HTTP REQUEST -----------
GET /bucket-name/?location= HTTP/1.1
Host: object-storage.nz-por-1.catalystcloud.io
User-Agent: MinIO (linux; amd64) minio-go/v7.0.5
Authorization: AWS4-HMAC-SHA256 Credential=[REDACTED]/20210523/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=9f526a92c25087d5b0ecfb9c6095c2988a3384564068883d4c01ceb7d6ff1fb0
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20210523T102711Z
Accept-Encoding: gzip


2021/05/23 22:27:11 debug/round_tripper_debug.go:95     debug.loggingRoundTripper.RoundTrip     38      ------------  HTTP RESPONSE ----------
HTTP/1.1 412 Precondition Failed
Content-Length: 7
Content-Type: text/html; charset=UTF-8
Date: Sun, 23 May 2021 10:27:11 GMT
Server: nginx/1.16.0
X-Trans-Id: txf28c0c958bd94268b698a-0060aa2dff


2021/05/23 22:27:11 backend/layout.go:132       backend.DetectLayout    1       layout detection failed
2021/05/23 22:27:11 backend/layout.go:156       backend.ParseLayout     1       error: auto-detecting the filesystem layout failed, use default layout default
2021/05/23 22:27:11 backend/layout.go:139       backend.ParseLayout     1       parse layout string "default" for backend at 
2021/05/23 22:27:11 debug/round_tripper_debug.go:82     debug.loggingRoundTripper.RoundTrip     1       ------------  HTTP REQUEST -----------
GET /bucket-name/?location= HTTP/1.1
Host: object-storage.nz-por-1.catalystcloud.io
User-Agent: MinIO (linux; amd64) minio-go/v7.0.5
Authorization: AWS4-HMAC-SHA256 Credential=[REDACTED]/20210523/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=9f526a92c25087d5b0ecfb9c6095c2988a3384564068883d4c01ceb7d6ff1fb0
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20210523T102711Z
Accept-Encoding: gzip


2021/05/23 22:27:11 debug/round_tripper_debug.go:95     debug.loggingRoundTripper.RoundTrip     1       ------------  HTTP RESPONSE ----------
HTTP/1.1 412 Precondition Failed
Content-Length: 7
Content-Type: text/html; charset=UTF-8
Date: Sun, 23 May 2021 10:27:11 GMT
Server: nginx/1.16.0
X-Trans-Id: tx5abb15255c674343b2751-0060aa2dff


2021/05/23 22:27:11 s3/s3.go:153        s3.(*Backend).IsAccessDenied    1       IsAccessDenied(minio.ErrorResponse, minio.ErrorResponse{XMLName:xml.Name{Space:"", Local:""}, Code:"PreconditionFailed", Message:"At least one of the pre-conditions you specified did not hold", BucketName:"bucket-name", Key:"", RequestID:"", HostID:"", Region:"", Server:"", StatusCode:412})
2021/05/23 22:27:11 s3/s3.go:136        s3.Create       1       BucketExists(bucket-name) returned err At least one of the pre-conditions you specified did not hold
2021/05/23 22:27:11 restic/lock.go:119  main.unlockAll  1       unlocking 0 locks

MichaelEischer · June 4, 2021, 10:46pm

You could try the configuration from S3 V2 api signature - #2 by MichaelEischer . Currently the only way to let restic use v2 signatures is via a minio client config file.

amuckart · June 27, 2021, 12:54am

Hi Micahel,

Thank you for that. I tried it and it didn’t work either. Since then the provider has upgraded and now supports v4 signatures, so I’ve gone back to using a more basic s3 config and I’m getting an almost identical error with a 412 precondition failed

I can’t replicate this with s3cmd. Using s3cmd and the same credentials I can list the buckets and contents, but restic can’t seem to.

2021/06/27 12:12:10 restic/main.go:95   main.main       1       main []string{"/usr/local/sbin/restic_0.12.0_linux_amd64_debugging", "init"}
2021/06/27 12:12:10 restic/main.go:96   main.main       1       restic 0.12.0 (v0.12.0-0-g27f24133) compiled with go1.13.8 on linux/amd64
2021/06/27 12:12:10 restic/global.go:751        main.create     1       parsing location s3:https://object-storage.nz-por-1.catalystcloud.io:443/servers
2021/06/27 12:12:10 restic/global.go:574        main.parseConfig        1       opening s3 repository at s3.Config{Endpoint:"object-storage.nz-por-1.catalystcloud.io:443", UseHTTP:false, KeyID:"", Secret:REDACTED
2021/06/27 12:12:10 s3/s3.go:38 s3.open 1       open, config s3.Config{Endpoint:"object-storage.nz-por-1.catalystcloud.io:443", UseHTTP:false, KeyID:"", Secret:REDACTED
2021/06/27 12:12:10 backend/layout.go:139       backend.ParseLayout     1       parse layout string "" for backend at 
2021/06/27 12:12:10 backend/layout.go:99        backend.DetectLayout    1       detect layout at 
2021/06/27 12:12:10 s3/s3.go:198        s3.(*Backend).ReadDir   1       ReadDir(keys)
2021/06/27 12:12:10 s3/s3.go:208        s3.(*Backend).ReadDir   1       using ListObjectsV1(false)
2021/06/27 12:12:10 debug/round_tripper_debug.go:82     debug.loggingRoundTripper.RoundTrip     6       ------------  HTTP REQUEST -----------
GET /servers/?location= HTTP/1.1
Host: object-storage.nz-por-1.catalystcloud.io
User-Agent: MinIO (linux; amd64) minio-go/v7.0.5
Accept-Encoding: gzip

2021/06/27 12:12:10 restic/main.go:95   main.main       1       main []string{"/usr/local/sbin/restic_0.12.0_linux_amd64_debugging", "init"}
2021/06/27 12:12:10 restic/main.go:96   main.main       1       restic 0.12.0 (v0.12.0-0-g27f24133) compiled with go1.13.8 on linux/amd64
2021/06/27 12:12:10 restic/global.go:751        main.create     1       parsing location s3:https://object-storage.nz-por-1.catalystcloud.io:443/servers
2021/06/27 12:12:10 restic/global.go:574        main.parseConfig        1       opening s3 repository at s3.Config{Endpoint:"object-storage.nz-por-1.catalystcloud.io:443", UseHTTP:false, KeyID:"", Secret:"[REDACTED]"
2021/06/27 12:12:10 s3/s3.go:38 s3.open 1       open, config s3.Config{Endpoint:"object-storage.nz-por-1.catalystcloud.io:443", UseHTTP:false, KeyID:"", Secret:"[REDACTED]"
2021/06/27 12:12:10 backend/layout.go:139       backend.ParseLayout     1       parse layout string "" for backend at 
2021/06/27 12:12:10 backend/layout.go:99        backend.DetectLayout    1       detect layout at 
2021/06/27 12:12:10 s3/s3.go:198        s3.(*Backend).ReadDir   1       ReadDir(keys)
2021/06/27 12:12:10 s3/s3.go:208        s3.(*Backend).ReadDir   1       using ListObjectsV1(false)
2021/06/27 12:12:10 debug/round_tripper_debug.go:82     debug.loggingRoundTripper.RoundTrip     6       ------------  HTTP REQUEST -----------
GET /servers/?location= HTTP/1.1
Host: object-storage.nz-por-1.catalystcloud.io
User-Agent: MinIO (linux; amd64) minio-go/v7.0.5
Accept-Encoding: gzip


2021/06/27 12:12:11 debug/round_tripper_debug.go:95     debug.loggingRoundTripper.RoundTrip     6       ------------  HTTP RESPONSE ----------
HTTP/1.1 412 Precondition Failed
Content-Length: 7
Content-Type: text/html; charset=UTF-8
Date: Sun, 27 Jun 2021 00:12:11 GMT
Server: nginx/1.14.0 (Ubuntu)
X-Openstack-Request-Id: tx7234e3ac371e49cd8dd6d-0060d7c25b
X-Trans-Id: tx7234e3ac371e49cd8dd6d-0060d7c25b


2021/06/27 12:12:11 s3/s3.go:198        s3.(*Backend).ReadDir   1       ReadDir(key)
2021/06/27 12:12:11 s3/s3.go:208        s3.(*Backend).ReadDir   1       using ListObjectsV1(false)
2021/06/27 12:12:11 debug/round_tripper_debug.go:82     debug.loggingRoundTripper.RoundTrip     146     ------------  HTTP REQUEST -----------
GET /servers/?location= HTTP/1.1
Host: object-storage.nz-por-1.catalystcloud.io
User-Agent: MinIO (linux; amd64) minio-go/v7.0.5
Accept-Encoding: gzip


2021/06/27 12:12:11 debug/round_tripper_debug.go:95     debug.loggingRoundTripper.RoundTrip     146     ------------  HTTP RESPONSE ----------
HTTP/1.1 412 Precondition Failed
Content-Length: 7
Content-Type: text/html; charset=UTF-8
Date: Sun, 27 Jun 2021 00:12:11 GMT
Server: nginx/1.14.0 (Ubuntu)
X-Openstack-Request-Id: tx84992aa367d9475ab7f46-0060d7c25b
X-Trans-Id: tx84992aa367d9475ab7f46-0060d7c25b


2021/06/27 12:12:11 backend/layout.go:132       backend.DetectLayout    1       layout detection failed
2021/06/27 12:12:11 backend/layout.go:156       backend.ParseLayout     1       error: auto-detecting the filesystem layout failed, use default layout default
2021/06/27 12:12:11 backend/layout.go:139       backend.ParseLayout     1       parse layout string "default" for backend at 
2021/06/27 12:12:11 debug/round_tripper_debug.go:82     debug.loggingRoundTripper.RoundTrip     1       ------------  HTTP REQUEST -----------
GET /servers/?location= HTTP/1.1
Host: object-storage.nz-por-1.catalystcloud.io
User-Agent: MinIO (linux; amd64) minio-go/v7.0.5
Accept-Encoding: gzip


2021/06/27 12:12:11 debug/round_tripper_debug.go:95     debug.loggingRoundTripper.RoundTrip     1       ------------  HTTP RESPONSE ----------
HTTP/1.1 412 Precondition Failed
Content-Length: 7
Content-Type: text/html; charset=UTF-8
Date: Sun, 27 Jun 2021 00:12:11 GMT
Server: nginx/1.14.0 (Ubuntu)
X-Openstack-Request-Id: tx710059721f4c4813b50c3-0060d7c25b
X-Trans-Id: tx710059721f4c4813b50c3-0060d7c25b


2021/06/27 12:12:11 s3/s3.go:153        s3.(*Backend).IsAccessDenied    1       IsAccessDenied(minio.ErrorResponse, minio.ErrorResponse{XMLName:xml.Name{Space:"", Local:""}, Code:"PreconditionFailed", Message:"At least one of the pre-conditions you specified did not hold", BucketName:"servers", Key:"", RequestID:"", HostID:"", Region:"", Server:"", StatusCode:412})
2021/06/27 12:12:11 s3/s3.go:136        s3.Create       1       BucketExists(servers) returned err At least one of the pre-conditions you specified did not hold
2021/06/27 12:12:11 restic/lock.go:119  main.unlockAll  1       unlocking 0 locks

MichaelEischer · June 29, 2021, 8:04pm

Juding from the debug log, it looks like the KeyID is not set. How exactly do you pass the S3 credentials to restic?