S3TransferManager SdkClientException: Failed to send the request: Retry cannot be attempted because the retry strategy has prevented

Describe the bug

Sometimes large files (1GB ) upload failed with SDKClientException “Failed to send the request: Retry cannot be attempted because the retry strategy has prevented”.

The S3 Transfer Manager is configured as:

    this.transferManager =
        S3TransferManager.builder()
            .s3ClientConfiguration(
                cfg ->
                    cfg.credentialsProvider(awsCredentialsProvider)
                        .region(Region.of(s3Region))
                        .targetThroughputInGbps(20.0)
                        .minimumPartSizeInBytes(5 * SizeConstant.MB))
                .build();

I didn’t prevent the retry strategy. I’m wondering if this is prevented by default.

Expected Behavior

Files could be uploaded successfully.

Current Behavior

Exceptions thrown:

Caused by: java.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException: Failed to send the request: Retry cannot be attempted because the retry strategy has prevented the operation.
	at software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:62)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:51)
	at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
	at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
	at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:76)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeAttemptExecute(AsyncRetryableStage.java:103)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:181)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:159)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
	at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:76)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$null$0(MakeAsyncHttpRequestStage.java:103)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:165)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
	at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	... 1 more
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Failed to send the request: Retry cannot be attempted because the retry strategy has prevented the operation.
	at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:102)
	at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:43)
	at software.amazon.awssdk.transfer.s3.internal.S3CrtResponseHandlerAdapter.handleError(S3CrtResponseHandlerAdapter.java:93)
	at software.amazon.awssdk.transfer.s3.internal.S3CrtResponseHandlerAdapter.onFinished(S3CrtResponseHandlerAdapter.java:73)
	at software.amazon.awssdk.crt.s3.S3MetaRequestResponseHandlerNativeAdapter.onFinished(S3MetaRequestResponseHandlerNativeAdapter.java:19)

Reproduction Steps

This issue may be caused by unstable network and is hard to reproduce.

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

2.17.191-PREVIEW

JDK version used

1.8.0_311

Operating System and version

macOS Monterey 12.5.1

[TingDaoK]

In C, we default to 10 Gbps, here. For CLI, we default to 5 Gbps here. I am not sure about Java, @zoewangg may know better.

It’s useful to set, like our test for best performance from c5n.18xlarge instance will set it 100. And for your use case in poor network, it’s probably better to set it to 1 or something.

[zoewangg]

@TingDaoK Is there any way we can improve the error message? The current message “Failed to send the request: Retry cannot be attempted because the retry strategy has prevented the operation.” does not provide much information. If CRT could return a specific error code for this case, we can add the guidance in the error message.