Timing issue when creating bucket right after user

The PutBucketPolicyRequest fails in our process at the first attempt with a Invalid principal in policy error. This seems like a timing issue because the referenced user is created right before the PutBucketPolicyRequest which succeeds after a couple of retries.

Exception in thread "vert.x-eventloop-thread-7" software.amazon.awssdk.services.s3.model.S3Exception: Invalid principal in policy (Service: S3, Status Code: 400, Request ID: 72DC1788979CCEFC, Extended Request ID: I2C3iXnFi259I235bZstN55k8dZBtQKwjLh/XStWEM17aNyS4Tui6A1UkTXenIxxASw2u+gpZxg=)
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156)
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:106)
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:84)
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:42)
	at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$4(BaseClientHandler.java:214)
	at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler.lambda$prepare$0(AsyncResponseHandler.java:88)
	at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1072)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
	at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler$BaosSubscriber.onComplete(AsyncResponseHandler.java:129)
	at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.runAndLogError(ResponseHandler.java:179)
	at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.access$500(ResponseHandler.java:69)
	at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler$PublisherAdapter$1.onComplete(ResponseHandler.java:295)
	at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher.publishMessage(HandlerPublisher.java:402)
	at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher.flushBuffer(HandlerPublisher.java:338)
	at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher.receivedDemand(HandlerPublisher.java:291)
	at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher.access$200(HandlerPublisher.java:61)
	at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher$ChannelSubscription$1.run(HandlerPublisher.java:495)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:497)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at java.base/java.lang.Thread.run(Thread.java:834)

Steps to Reproduce

1. Create IAM User

val createUserRequest = CreateUserRequest.builder().userName("my-user").build()
val user = iamClient.createUser(createUserRequest).await().user()

2. Create S3 Bucket with Private Canned ACL

val createBucketRequest = CreateBucketRequest.builder()
            .bucket("my-bucket")
            .acl(BucketCannedACL.PRIVATE).build()

s3Client.createBucket(createBucketRequest).await()

3. Grant Access to the Bucket using a Policy

val policy = """ 
                {
                   "Version":"2012-10-17",
                   "Statement":[
                      {
                         "Sid":"AllowObjectReadWrite",
                         "Effect":"Allow",
                         "Principal":{
                            "AWS":"${user.userId()}"
                         },
                         "Action":[
                            "s3:GetObject",
                            "s3:PutObject"
                         ],
                         "Resource":[
                            "arn:aws:s3:::my-bucket/*"
                         ]
                      },
                      {
                         "Sid":"AllowBucketList",
                         "Effect":"Allow",
                         "Principal":{
                            "AWS":"${user.userId()}"
                         },
                         "Action":[
                            "s3:ListBucket"
                         ],
                         "Resource":[
                            "arn:aws:s3:::my-bucket"
                         ]
                      }
                   ]
                }
            """.trimIndent()

val policyRequest = PutBucketPolicyRequest.builder()
    .bucket(bucketName(system))
    .policy(policy)
    .build()

return s3Client.putBucketPolicy(policyRequest).await()

Workaround

The third, step which contains the PutBucketPolicyRequest, succeeds if we use a retry mechanism.

retry(limitAttempts(5) + constantDelay(5_000)) { s3Client.putBucketPolicy(policyRequest).await() }

Context

We’re working on a cloud platform that manages parking systems. A business requirement is that each parking system can stream its local database backups to a s3 bucket. In order to accomplish that goal a user and a bucket is created for each parking system during the provisioning process.

Your Environment

• AWS Java SDK version used: 2.13.51
• JDK version used: 11.0.2
• Operating System and version: macOS Mojave 10.14.6

[debora-ito]

If the userExists waiter is working, one possibility is that by the time you set the bucket policy the user creation hasn’t been propagated yet, as one of the referenced docs seems to support:

Add wait time between subsequent commands, even if a Describe command returns an accurate response. Apply an exponential backoff algorithm starting with a couple of seconds of wait time, and increase gradually up to about five minutes of wait time.

The behavior you described in the original question is expected in the AWS infrastructure model, and retrying is one way to manage it.

[debora-ito]

Hi @deen13 every resource change can take some time to propagate, it takes time for the information to be reflected globally in the AWS infrastructure. The solution for this issue is Waiters but they are not available yet in Java v2, we are tracking the feature request in #24 (the linked blog post is for Java v1).

Let us know if that helps.