NettyNioAsyncHttpClient not respecting replacements of JSSE Providers (e.g. Bouncy Castle FIPS) by using own CipherSuites

Hey everyone! I am currently setting up the AWS SDK to run with SQS and S3. However my service is running in a FIPS-compliant, standardised Docker environment. Unfortunately, this also means exchanging used Security Providers (OpenSSL is excluded from the classpath through Gradle) under the Hood of JVM and thus also the SDK.

The problem occurs when trying to spin up an SdkAsyncHttpClient with a corresponding NettyNioAsyncHttpClient as I am running in Spring Boot with Netty already for other operations and I want to keep things consistend and non-blocking.

Describe the bug

When trying to establish the connection via Netty in the SDK to AWS Services, Ciphers cannot be negotiated correctly and the SSL Handshake fails.

{"@timestamp": "2020-11-23T16:28:40.471+0000", "level": "INFO", "class": "org.bouncycastle.jsse.provider.ProvTlsServer", "description": "Server raised fatal(2) handshake_failure(40) alert: Failed to process record org.bouncycastle.tls.TlsFatalAlert: handshake_failure(40) 
at org.bouncycastle.tls.AbstractTlsServer.getSelectedCipherSuite(Unknown Source) 
at org.bouncycastle.jsse.provider.ProvTlsServer.getSelectedCipherSuite(Unknown Source) 
at org.bouncycastle.tls.TlsServerProtocol.generateServerHelloMessage(Unknown Source) 
at org.bouncycastle.tls.TlsServerProtocol.handleHandshakeMessage(Unknown Source)}

Expected Behavior

I was expecting a successful handshake with the AWS Server performed with using Cipher Suites common to both the Bouncy Castle FIPS Library responsible for TLS (JSSE) through Netty which under the Hood usually defaults to the correct SSL Settings if configured through the JVM.

Whenever trying to establish a connection, a cipher that is compatible should be negotiated and the TLS Handshake completed. I can replicate a successful handshake with a Netty Client spun up by myself in my service and negotiating with a different non-aws service (e.g. A MySQL through Hikari):

Client notified of selected protocol version: TLSv1.2 Client notified of selected cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384

I am using the same default SSL Context and Provider that I am expecting to be drawn in to the SDK from the JVM as well.

Current Behavior

in the case of the above failure, I was accessing us-west-2.queue.amazonaws.com

According to the SSL Analysis here: https://www.ssllabs.com/ssltest/analyze.html?d=us-west-2.queue.amazonaws.com

and my SSL Configuration below

Protocols SSLContext.getDefault().getDefaultSSLParameters().getProtocols() [TLSv1.2, TLSv1.1, TLSv1] CipherSuites SSLContext.getDefault().getDefaultSSLParameters().getCipherSuites() [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA]

From my limited knowledge (I am not exactly a security expert), there should be several ciphers listed that are supported for both my client and server for negotiation, indicating an issue in the SDK

What I am seeing however, is the following: [id: OMITTED] REGISTERED [id: OMITTED] CONNECT Client raised warning(1) user_canceled(90) alert: User canceled handshake Client raised warning(1) close_notify(0) alert: Connection closed [id: OMITTED, L:/OMITTED- R:sqs.us-west-2.amazonaws.com/OMITTED:443] USER_EVENT: SslHandshakeCompletionEvent(java.lang.IllegalStateException: No usable cipher suites enabled)

Steps to Reproduce

The JVM is setup like this in a Dockerfile

ENV JAVA_OPTS --show-version \ -XshowSettings:vm \ -XX:+DisableExplicitGC \ -Dfile.encoding=UTF-8 \ -Dorg.bouncycastle.fips.approved_only=true \ -Xbootclasspath/a:/PATHTOJARS/bc-fips-${BC_VERSION}.jar \ -Xbootclasspath/a:/PATHTOJARS/bctls-fips-${BC_TLS_VERSION}.jar"

I am using these Options to generate my Keystore

ENV KEYTOOL_OPTS "-storetype BCFKS \ -providerpath /PATHTOJARS/bc-fips-${BC_VERSION}.jar \ -providerclass org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider"

and some changes to the java.security file for setting up the JVM correctly.

RUN sed -i 's/\(security\.provider\.[0-9]\+\)=\(.*\)/##\1=\2/g' java.security
RUN sed -i 's/\(ssl\.KeyManagerFactory\.algorithm\)=.*/\1=PKIX/g' java.security
RUN sed -i 's/\(securerandom\.source\)=.*/\1=file:\/dev\/urandom/g' java.security
RUN sed -i 's/\(securerandom\.strongAlgorithms\)=.*/\1=NativePRNGNonBlocking:SUN,DRBG:SUN/g' java.security
RUN printf '\nsecurity.provider.1=BCFIPS\nsecurity.provider.2=BCJSSE fips:BCFIPS\nsecurity.provider.3=SUN\n' >> java.security

as well as

-Dio.netty.handler.ssl.noOpenSsl=true \ -Djavax.net.ssl.trustStoreProvider=BCFIPS \ -Djavax.net.ssl.trustStoreType=BCFKS \ -Djavax.net.ssl.trustStorePassword=changeit \ -Djavax.net.debug=ssl,handshake \

If you want to configure the providers by Code you can use this:

compile group: 'org.bouncycastle', name: 'bc-fips', version: '1.0.2'
compile group: 'org.bouncycastle', name: 'bctls-fips', version: '1.0.10'

private static void makeOurServiceFIPSCompliant() {
        Security.insertProviderAt(new BouncyCastleFipsProvider(),1); // basic encryption provider
        Security.insertProviderAt(new BouncyCastleJsseProvider(),2); // tls
        Security.removeProvider("SunRsaSign");
        Security.removeProvider("SunJSSE");
    }

for your Runtime you can probably replicate the issue just like I have. It is important to fully change out the backing security implementation and also access Netty with JDK and not OpenSSL. Due to security concerns, I will not be able to share the full Dockerfile with you but please mention if there is anything in particular you are interested in.

Possible Solution

Support native readout of Netty Security Context (SSLContext) and thus usability of an adapted JSSE through the JDK Provider instead of OpenSSL. Unfortunately I am lacking the insight on where this is currently problematic or wether I am missing a configuration of some sort.

Context

I am trying to achieve full FIPS compliance through AWS and am currently failing at achieving full compliancy due to the Handshake error.

Your Environment

• AWS Java SDK version used: 2.15.34 implementation("software.amazon.awssdk:s3:${awsSdkVersion}") api("software.amazon.awssdk:sqs:${awsSdkVersion}")

• JDK version used: openjdk 11.0.9 2020-10-20 OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.9+11) OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.9+11, mixed mode)

• Operating System and version: Red Hat Enterprise Linux 7

• Additional Info: ARG BC_VERSION=1.0.2 ARG BC_TLS_VERSION=1.0.10

[cenedhryn]

@jakobmoellersap We’ll add support for FIPS compliant algorithms to the Netty client and are tracking this issue internally.

[github-actions[bot]]

It looks like this issue hasn’t been active in longer than a week. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please add a comment to prevent automatic closure, or if the issue is already closed please feel free to reopen it.