Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Using Custom HTTP client impl with SQSClient causes AWS4 signature generation to fail

See original GitHub issue

Describe the bug

I am using a custom HTTP client implementation (SdkHttpClient) that uses Java 11’s java.net.http.HttpClient. The implementation is quite straight-forward:

public class JavaHttpClient implements SdkHttpClient {
    private static final String CLIENT_NAME = "JavaNetHttpClient";
    private final HttpClient httpClient;

    private JavaHttpClient(AttributeMap options) {
        httpClient = HttpClient.newBuilder()
                .connectTimeout(options.get(CONNECTION_TIMEOUT))
                .version(options.get(PROTOCOL) == HTTP2 ? HTTP_2 : HTTP_1_1)
                .build();
    }

    public static Builder builder() {
        return new DefaultBuilder();
    }

    /**
     * Create a {@link HttpClient} client with the default properties.
     *
     * @return a {@link JavaHttpClient}
     */
    public static SdkHttpClient create() {
        return new DefaultBuilder().build();
    }

    @Override
    public ExecutableHttpRequest prepareRequest(HttpExecuteRequest request) {
        HttpRequest.Builder requestBuilder = HttpRequest.newBuilder().uri(request.httpRequest().getUri());
        for (Map.Entry<String, List<String>> header : request.httpRequest().headers().entrySet()) {
            // avoid java.lang.IllegalArgumentException: restricted header names
            if (!header.getKey().equalsIgnoreCase("Content-Length") && !header.getKey().equalsIgnoreCase("Host")) {
                for (String headerVal : header.getValue()) {
                    requestBuilder = requestBuilder.header(header.getKey(), headerVal);
                }
            }
        }
        requestBuilder = switch (request.httpRequest().method()) {
            case POST -> requestBuilder.POST(HttpRequest.BodyPublishers.ofInputStream(
                    () -> request.contentStreamProvider().orElseThrow().newStream()));
            case PUT -> requestBuilder.PUT(HttpRequest.BodyPublishers.ofInputStream(
                    () -> request.contentStreamProvider().orElseThrow().newStream()));
            case DELETE -> requestBuilder.DELETE();
            case HEAD -> requestBuilder.method("HEAD", HttpRequest.BodyPublishers.noBody());
            case PATCH -> throw new UnsupportedOperationException("PATCH not supported");
            case OPTIONS -> requestBuilder.method("OPTIONS", HttpRequest.BodyPublishers.noBody());
            default -> requestBuilder;
        };
        return new RequestCallable(httpClient, requestBuilder.build());
    }

    @Override
    public void close() {}

    @Override
    public String clientName() {
        return CLIENT_NAME;
    }

    private static class RequestCallable implements ExecutableHttpRequest {
        final HttpClient httpClient;
        final HttpRequest request;

        private RequestCallable(HttpClient httpClient, HttpRequest request) {
            this.httpClient = httpClient;
            this.request = request;
        }

        @Override
        public HttpExecuteResponse call() throws IOException {
            HttpResponse<InputStream> response;
            try {
                response = httpClient.send(request, HttpResponse.BodyHandlers.ofInputStream());
            } catch (InterruptedException ex) {
                ex.printStackTrace();
                throw new IOException(ex);
            }

            return HttpExecuteResponse.builder()
                    .response(SdkHttpResponse.builder()
                            .statusCode(response.statusCode())
                            .headers(response.headers().map())
                            .build())
                    .responseBody(AbortableInputStream.create(response.body()))
                    .build();
        }

        @Override
        public void abort() {}
    }

    private static final class DefaultBuilder implements Builder {
        private final AttributeMap.Builder standardOptions = AttributeMap.builder();

        private DefaultBuilder() {}

        /**
         * Sets the read timeout to a specified timeout. A timeout of zero is interpreted as an infinite timeout.
         *
         * @param socketTimeout the timeout as a {@link Duration}
         * @return this object for method chaining
         */
        public Builder socketTimeout(Duration socketTimeout) {
            standardOptions.put(READ_TIMEOUT, socketTimeout);
            return this;
        }

        public void setSocketTimeout(Duration socketTimeout) {
            socketTimeout(socketTimeout);
        }

        /**
         * Sets the connect timeout to a specified timeout. A timeout of zero is interpreted as an infinite timeout.
         *
         * @param connectionTimeout the timeout as a {@link Duration}
         * @return this object for method chaining
         */
        public Builder connectionTimeout(Duration connectionTimeout) {
            standardOptions.put(CONNECTION_TIMEOUT, connectionTimeout);
            return this;
        }

        public void setConnectionTimeout(Duration connectionTimeout) {
            connectionTimeout(connectionTimeout);
        }

        public Builder protocol(Protocol protocol) {
            standardOptions.put(PROTOCOL, protocol);
            return this;
        }

        /**
         * Used by the SDK to create a {@link SdkHttpClient} with service-default values if no other values have been
         * configured.
         *
         * @param serviceDefaults Service specific defaults. Keys will be one of the constants defined in
         * {@link SdkHttpConfigurationOption}.
         * @return an instance of {@link SdkHttpClient}
         */
        @Override
        public SdkHttpClient buildWithDefaults(AttributeMap serviceDefaults) {
            return new JavaHttpClient(standardOptions.build()
                    .merge(serviceDefaults)
                    .merge(SdkHttpConfigurationOption.GLOBAL_HTTP_DEFAULTS));
        }
    }
}

I have successfully used this client with DynamoDB thusly:

SdkHttpClient awsSdkHttpClient = JavaHttpClient.create();
dynamoDbClient = DynamoDbClient.builder()
		.region(Region.US_WEST_2)
		.httpClient(awsSdkHttpClient).build();

However, attempting to use this SdkHttpClient implementation with SQS yields the following exception:

software.amazon.awssdk.services.sqs.model.SqsException: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

It seems strange to me that the same HTTP client would work with one service (DynamoDB) but fail with another (SQS) which leads me to believe something may be bugged on the AWS side of things.

Thank you.

Expected Behavior

SQSClient and DynamoDB client both work correctly with the custom SdkHttpClient implementation.

Current Behavior

Only DynamoDB works correctly with the custom SdkHttpClient implementation. The SQSClient seems to create the wrong AWS4 signature verification.

Reproduction Steps

public class JavaHttpClient implements SdkHttpClient {
    private static final String CLIENT_NAME = "JavaNetHttpClient";
    private final HttpClient httpClient;

    private JavaHttpClient(AttributeMap options) {
        httpClient = HttpClient.newBuilder()
                .connectTimeout(options.get(CONNECTION_TIMEOUT))
                .version(options.get(PROTOCOL) == HTTP2 ? HTTP_2 : HTTP_1_1)
                .build();
    }

    public static Builder builder() {
        return new DefaultBuilder();
    }

    /**
     * Create a {@link HttpClient} client with the default properties.
     *
     * @return a {@link JavaHttpClient}
     */
    public static SdkHttpClient create() {
        return new DefaultBuilder().build();
    }

    @Override
    public ExecutableHttpRequest prepareRequest(HttpExecuteRequest request) {
        HttpRequest.Builder requestBuilder = HttpRequest.newBuilder().uri(request.httpRequest().getUri());
        for (Map.Entry<String, List<String>> header : request.httpRequest().headers().entrySet()) {
            // avoid java.lang.IllegalArgumentException: restricted header names
            if (!header.getKey().equalsIgnoreCase("Content-Length") && !header.getKey().equalsIgnoreCase("Host")) {
                for (String headerVal : header.getValue()) {
                    requestBuilder = requestBuilder.header(header.getKey(), headerVal);
                }
            }
        }
        requestBuilder = switch (request.httpRequest().method()) {
            case POST -> requestBuilder.POST(HttpRequest.BodyPublishers.ofInputStream(
                    () -> request.contentStreamProvider().orElseThrow().newStream()));
            case PUT -> requestBuilder.PUT(HttpRequest.BodyPublishers.ofInputStream(
                    () -> request.contentStreamProvider().orElseThrow().newStream()));
            case DELETE -> requestBuilder.DELETE();
            case HEAD -> requestBuilder.method("HEAD", HttpRequest.BodyPublishers.noBody());
            case PATCH -> throw new UnsupportedOperationException("PATCH not supported");
            case OPTIONS -> requestBuilder.method("OPTIONS", HttpRequest.BodyPublishers.noBody());
            default -> requestBuilder;
        };
        return new RequestCallable(httpClient, requestBuilder.build());
    }

    @Override
    public void close() {}

    @Override
    public String clientName() {
        return CLIENT_NAME;
    }

    private static class RequestCallable implements ExecutableHttpRequest {
        final HttpClient httpClient;
        final HttpRequest request;

        private RequestCallable(HttpClient httpClient, HttpRequest request) {
            this.httpClient = httpClient;
            this.request = request;
        }

        @Override
        public HttpExecuteResponse call() throws IOException {
            HttpResponse<InputStream> response;
            try {
                response = httpClient.send(request, HttpResponse.BodyHandlers.ofInputStream());
            } catch (InterruptedException ex) {
                ex.printStackTrace();
                throw new IOException(ex);
            }

            return HttpExecuteResponse.builder()
                    .response(SdkHttpResponse.builder()
                            .statusCode(response.statusCode())
                            .headers(response.headers().map())
                            .build())
                    .responseBody(AbortableInputStream.create(response.body()))
                    .build();
        }

        @Override
        public void abort() {}
    }

    private static final class DefaultBuilder implements Builder {
        private final AttributeMap.Builder standardOptions = AttributeMap.builder();

        private DefaultBuilder() {}

        /**
         * Sets the read timeout to a specified timeout. A timeout of zero is interpreted as an infinite timeout.
         *
         * @param socketTimeout the timeout as a {@link Duration}
         * @return this object for method chaining
         */
        public Builder socketTimeout(Duration socketTimeout) {
            standardOptions.put(READ_TIMEOUT, socketTimeout);
            return this;
        }

        public void setSocketTimeout(Duration socketTimeout) {
            socketTimeout(socketTimeout);
        }

        /**
         * Sets the connect timeout to a specified timeout. A timeout of zero is interpreted as an infinite timeout.
         *
         * @param connectionTimeout the timeout as a {@link Duration}
         * @return this object for method chaining
         */
        public Builder connectionTimeout(Duration connectionTimeout) {
            standardOptions.put(CONNECTION_TIMEOUT, connectionTimeout);
            return this;
        }

        public void setConnectionTimeout(Duration connectionTimeout) {
            connectionTimeout(connectionTimeout);
        }

        public Builder protocol(Protocol protocol) {
            standardOptions.put(PROTOCOL, protocol);
            return this;
        }

        /**
         * Used by the SDK to create a {@link SdkHttpClient} with service-default values if no other values have been
         * configured.
         *
         * @param serviceDefaults Service specific defaults. Keys will be one of the constants defined in
         * {@link SdkHttpConfigurationOption}.
         * @return an instance of {@link SdkHttpClient}
         */
        @Override
        public SdkHttpClient buildWithDefaults(AttributeMap serviceDefaults) {
            return new JavaHttpClient(standardOptions.build()
                    .merge(serviceDefaults)
                    .merge(SdkHttpConfigurationOption.GLOBAL_HTTP_DEFAULTS));
        }
    }
}

    private static SqsClient getSqsClient() {
        if (sqsClient == null) {
            SdkHttpClient awsSdkHttpClient = JavaHttpClient.create();
            sqsClient = SqsClient.builder()
                    .httpClient(awsSdkHttpClient)
                    .region(Region.US_WEST_2).build();
        }
        return sqsClient;
    }

 // Try sending a message, or doing anything, with getSqsClient().

Here’s the full stack trace:

5dbbb75980d524adc6bf8b3724ab8520c2d2d853599058c4d76f3e0ae9d319be'
(Service: Sqs, Status Code: 403, Request ID: 28a6e9af-d842-539e-a9be-b36825348979, Extended Request ID: null)
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleErrorResponse(CombinedResponseHandler.java:125)
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:82)
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:60)
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:41)
at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40)
at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78)
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:40)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:81)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56)
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:167)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:82)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:175)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76)
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56)
at software.amazon.awssdk.services.sqs.DefaultSqsClient.sendMessage(DefaultSqsClient.java:1528)

Looking at the difference in the Authorization header between SQSClient and DynamoDBClient I do see one difference:

DynamoDB: [AWS4-HMAC-SHA256 Credential=[snip]/20220407/us-west-2/dynamodb/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date;x-amz-target, Signature=[snip]]

SQS: [AWS4-HMAC-SHA256 Credential=[snip]/20220407/us-west-2/sqs/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date, Signature=[snip]]

The difference is DynamoDB signs the x-amz-target header, whereas SQS doesn’t. However, I don’t think that’s the problem because SQS doesn’t use the x-amz-target header to my knowledge.

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

2.17.112

JDK version used

17

Operating System and version

Windows 10

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:7 (2 by maintainers)

github_iconTop GitHub Comments

1reaction
debora-itocommented, Apr 7, 2022

@brcolow nice work on customizing the Java 11 HttpClient.

Although I cannot help debug your code, I can give some pointers to help you identify where’s the problem.

I would enable the verbose wirelogs and compare a SQS call made with the SDK native Apache httpclient vs. a SQS call made with the JavaHttpClient. The verbose wirelogs will show the request headers and contents. Comparing a DynamoDB call with SQS call is not very helpful, different services work with different headers. For details on how to enable wirelogs on the SDK see: https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/logging-slf4j.html#sdk-java-logging-verbose

The other thing that can impact the signature is special characters, the SDK url-encodes values by default, so this might be worth checking.

Let us know what you find.

0reactions
brcolowcommented, Apr 8, 2022

I did notice one thing when trying to debug this. The SQS message is being sent via a POST body, and according to the headers passed by AWS to the java.net.http client, the Content-Length is 12638 bytes. But when the error happens and AWS says what the canonical request should be, it says:

The Canonical String for this request should have been
'POST
/

amz-sdk-invocation-id:41fdd492-0fa2-664f-b514-52f06cd798a4
amz-sdk-request:attempt=1; max=4
content-length:
content-type:application/x-www-form-urlencoded; charset=utf-8
host:sqs.us-west-2.amazonaws.com
x-amz-date:20220408T211843Z

It seems like the java.net.http client is not setting Content-Length header. But, this is a restricted header that java.net.http doesn’t allow users to manually set. Very strange…

Read more comments on GitHub >

github_iconTop Results From Across the Web

Signing AWS API requests - AWS General Reference
When an AWS service receives an authenticated request, it recreates the signature using the authentication information contained in the request.
Read more >
How to generate Signature in AWS from Java - Stack Overflow
can I use aws-v4-signer for generating a signature (x-amz-signature) for an s3 form post request? I believe no headers are needed in this...
Read more >
AWS Simple Queue Service (SQS) - Apache Camel
Name Description Default amazonAWSHost (common) The hostname of the Amazon AWS cloud. amazonaws.com amazonSQSClient (common) Autowired To use the AmazonSQS as client. autoCreateQueue (common) Setting the...
Read more >
Issue with generating "AWS Signature Version 4" fo...
Note: Amazon V4 signature based authentication can also be used from Script background. So use it in is my script:
Read more >
class SQS. Client - Boto3 Docs 1.26.28 documentation
If you reach this limit, Amazon SQS returns the OverLimit error message. To avoid reaching the limit, you should delete messages from the...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

EventBridge PutEvent Successful with Invalid EventBridge ARN or EventBus name

Next page

Minimum TTL on instance profile credentials results in unstable connections to AWS resources