error 403 in transfer manager when upload big file

Transfer manager is returning a 403 error when trying to upload files bigger than 800MB.

Describe the bug

In a multi-module project the parent pom is configured like this:

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>bom</artifactId>
            <version>2.17.34</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency> 
</dependencyManagement>

and the child pom dependencies are:

<dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>s3</artifactId>
            <exclusions>
                <exclusion>
                    <groupId>software.amazon.awssdk</groupId>
                    <artifactId>netty-nio-client</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>aws-crt-client</artifactId>
            <version>2.17.34-PREVIEW</version>
        </dependency>
        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>s3-transfer-manager</artifactId>
            <version>2.17.34-PREVIEW</version>
        </dependency>

So default crt http client is used. The implementation of the upload code:

S3TransferManager transferManager = S3TransferManager.builder()
                    .s3ClientConfiguration(b -> b.credentialsProvider( StaticCredentialsProvider.create(AwsBasicCredentials.create(this.key, this.secret) ))                       
                            .region(Region.of(this.region)))
                    .build();

            Upload upload = transferManager.upload(
                    b -> b.source(file.toPath()).putObjectRequest(r -> r.bucket(bucket).key(finalKey))
            );
            CompletedUpload completedUpload =
                     upload.completionFuture().join();

            System.out.println("PutObjectResponse: " + completedUpload.response());

Several configurations of maxConcurrency, minimumPartSizeInBytes and targetThroughputInGbps has been tested with same results.

The related policy is full access to s3 resources:

{
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": "*"
        }
    ]
}

Expected Behavior

The target file is upoloaded to the bucket.

Current Behavior

Several files has been tested (zip, avi and pdf formats). Files with size lower than 800MB uploads fine. Bigger files throws a 403 error:

java.util.concurrent.CompletionException: software.amazon.awssdk.services.s3.model.S3Exception: Invalid response status from request

	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
	at software.amazon.awssdk.transfer.s3.internal.DefaultS3CrtAsyncClient.lambda$putObject$3(DefaultS3CrtAsyncClient.java:137)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
	at com.amazonaws.s3.S3NativeClient$3.onFinished(S3NativeClient.java:271)
	at software.amazon.awssdk.crt.s3.S3MetaRequestResponseHandlerNativeAdapter.onFinished(S3MetaRequestResponseHandlerNativeAdapter.java:19)
Caused by: software.amazon.awssdk.services.s3.model.S3Exception: Invalid response status from request
	at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:95)
	at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:55)
	at software.amazon.awssdk.transfer.s3.internal.CrtErrorHandler.getServiceSideException(CrtErrorHandler.java:71)
	at software.amazon.awssdk.transfer.s3.internal.CrtErrorHandler.lambda$transformException$0(CrtErrorHandler.java:55)
	at java.util.Optional.map(Optional.java:215)
	at software.amazon.awssdk.transfer.s3.internal.CrtErrorHandler.transformException(CrtErrorHandler.java:55)
	at software.amazon.awssdk.transfer.s3.internal.DefaultS3CrtAsyncClient.lambda$putObject$3(DefaultS3CrtAsyncClient.java:138)
	... 6 more
Caused by: software.amazon.awssdk.crt.s3.CrtS3RuntimeException: Invalid response status from request AWS_ERROR_S3_INVALID_RESPONSE_STATUS(9223)
	at com.amazonaws.s3.S3NativeClient$3.onFinished(S3NativeClient.java:263)
	... 1 more

Steps to Reproduce

Use the configuration and code described above.

Possible Solution

Context

Your Environment

• AWS Java SDK version used: 2.17.34
• JDK version used: jdk1.8.0_181
• Operating System and version: Manjaro Linux. Kernel 5.9.16-1

[chipironVacilon]

@debora-ito thanks for the logger info. It will be quite useful. I spend several hours trying to make it work following your first indications XD. @zoewangg New transfer manager works faster and is easier to implement. Quite useful for me. I write applications that makes massive uploads to S3 so processes has been boosted. I´m happy with this 😃

[chipironVacilon]

@zoewangg That’s amazing! Thanks for your great job. I will give it a try in the next few days.