SSL Handshake failure in Android 10

Android 10 throws exception in SSL Handshaking both in emulators and Pixel devices. The same code works for Android 9 and before. Steps to reproduce:

1. Create a Private Public Key Pair. KeyPairGenerator kpg = KeyPairGenerator.getInstance( KeyProperties.KEY_ALGORITHM_RSA);

        kpg.initialize(new KeyGenParameterSpec.Builder(
                CLIENT_KEY_ALIAS,
                KeyProperties.PURPOSE_SIGN | KeyProperties.PURPOSE_VERIFY)
                .setDigests(KeyProperties.DIGEST_NONE)
                .setSignaturePaddings(KeyProperties.SIGNATURE_PADDING_RSA_PKCS1)
                .build());
   
        KeyPair keyPair = kpg.generateKeyPair();
        return keyPair;
   

2. Send public key to server and receives client certiricates, and store it in Android Keystore. As well as store Root Server certificate.

        CertificateFactory certificateFactory = CertificateFactory.getInstance("X.509");
   
        X509Certificate[] clientCertArray = new X509Certificate[clientCertArrayFinal.length];
        i = 0;
   
        for (String indCert : clientCertArrayFinal) {
            X509Certificate clientCertFromApi = (X509Certificate) certificateFactory.generateCertificate(new
            ByteArrayInputStream(indCert.getBytes()));
            clientCertArray[i++] = clientCertFromApi;
        }
   
        PrivateKey privateKeyClient = iPrivateKeyClient;
   
        InputStream inServer = iContext.getResources().openRawResource(R.raw.mrca);
        X509Certificate serverCert = (X509Certificate) certificateFactory.generateCertificate(inServer);
   
        KeyStore androidKeyStore = KeyStore.getInstance(CertificateManager.ANDROID_KEYSTORE);
        androidKeyStore.load(null);
   
        androidKeyStore.setCertificateEntry(CertificateManager.SERVER_CERT_ALIAS, serverCert);  //Server cert
        androidKeyStore.setKeyEntry(CertificateManager.CLIENT_KEY_ALIAS, privateKeyClient, null, clientCertArray); //Client cert, can be opened using private key
   

3. Using Trustmananger to create SSLContext

        KeyStore androidKeyStore = KeyStore.getInstance(CertificateManager.ANDROID_KEYSTORE);
        androidKeyStore.load(null);
   
        TrustManagerFactory trustmanagerfactory =
       TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
        trustmanagerfactory.init(androidKeyStore);
        TrustManager[] trustManagers = trustmanagerfactory.getTrustManagers();
        KeyManagerFactory kmf = KeyManagerFactory
                .getInstance(KeyManagerFactory.getDefaultAlgorithm());
        kmf.init(androidKeyStore, "".toCharArray());
   
        final SSLContext sslContext = SSLContext.getInstance("TLS");
        sslContext.init(kmf.getKeyManagers(), trustManagers, null);
   

4. Start handshake with server: String server = “185.58.87.115”; SSLSocketFactory ssf = sslContext.getSocketFactory();

                    InetSocketAddress inetSocketAddress = new InetSocketAddress(server, 443);
   
                    SSLSocket socket = (SSLSocket) ssf.createSocket();
                    socket.connect(inetSocketAddress);
                    socket.setKeepAlive(true);
   
                    socket.addHandshakeCompletedListener(new HandshakeCompletedListener() {
                        @Override
                        public void handshakeCompleted(HandshakeCompletedEvent event) {
                            try {
                                String principal = event.getPeerPrincipal().getName();
                                Log.i(TAG, principal);
                                Log.i(TAG, event.getCipherSuite().toString());
   
                            } catch (SSLPeerUnverifiedException e) {
                                e.printStackTrace();
                            }
   
                        }
                    });
                    Thread.sleep(1000);
                    socket.startHandshake();      
   

The handshake failed with below exception:

`W/CryptoUpcalls: Preferred provider doesn’t support key: java.security.InvalidKeyException: Keystore operation failed at android.security.KeyStore.getInvalidKeyException(KeyStore.java:1362) at android.security.KeyStore.getInvalidKeyException(KeyStore.java:1402) at android.security.keystore.KeyStoreCryptoOperationUtils.getInvalidKeyExceptionForInit(KeyStoreCryptoOperationUtils.java:54) at android.security.keystore.KeyStoreCryptoOperationUtils.getExceptionForCipherInit(KeyStoreCryptoOperationUtils.java:89) at android.security.keystore.AndroidKeyStoreCipherSpiBase.ensureKeystoreOperationInitialized(AndroidKeyStoreCipherSpiBase.java:265) at android.security.keystore.AndroidKeyStoreCipherSpiBase.engineInit(AndroidKeyStoreCipherSpiBase.java:109) at javax.crypto.Cipher.tryTransformWithProvider(Cipher.java:2984) at javax.crypto.Cipher.tryCombinations(Cipher.java:2891) at javax.crypto.Cipher$SpiAndProviderUpdater.updateAndGetSpiAndProvider(Cipher.java:2796) at javax.crypto.Cipher.chooseProvider(Cipher.java:773) at javax.crypto.Cipher.init(Cipher.java:1143) at javax.crypto.Cipher.init(Cipher.java:1084) at com.android.org.conscrypt.CryptoUpcalls.rsaOpWithPrivateKey(CryptoUpcalls.java:173) at com.android.org.conscrypt.CryptoUpcalls.rsaSignDigestWithPrivateKey(CryptoUpcalls.java:132) at com.android.org.conscrypt.NativeCrypto.SSL_do_handshake(Native Method) at com.android.org.conscrypt.NativeSsl.doHandshake(NativeSsl.java:387) at com.android.org.conscrypt.ConscryptFileDescriptorSocket.startHandshake(ConscryptFileDescriptorSocket.java:226) at asynctls.mimecast.com.asynctlsapp.MainActivity$1.run(MainActivity.java:352) at java.lang.Thread.run(Thread.java:919) Caused by: android.security.KeyStoreException: Incompatible padding mode at android.security.KeyStore.getKeyStoreException(KeyStore.java:1292) at android.security.KeyStore.getInvalidKeyException(KeyStore.java:1402) at android.security.keystore.KeyStoreCryptoOperationUtils.getInvalidKeyExceptionForInit(KeyStoreCryptoOperationUtils.java:54) at android.security.keystore.KeyStoreCryptoOperationUtils.getExceptionForCipherInit(KeyStoreCryptoOperationUtils.java:89) at android.security.keystore.AndroidKeyStoreCipherSpiBase.ensureKeystoreOperationInitialized(AndroidKeyStoreCipherSpiBase.java:265) at android.security.keystore.AndroidKeyStoreCipherSpiBase.engineInit(AndroidKeyStoreCipherSpiBase.java:109) at javax.crypto.Cipher.tryTransformWithProvider(Cipher.java:2984) at javax.crypto.Cipher.tryCombinations(Cipher.java:2891) at javax.crypto.Cipher$SpiAndProviderUpdater.updateAndGetSpiAndProvider(Cipher.java:2796) at javax.crypto.Cipher.chooseProvider(Cipher.java:773) at javax.crypto.Cipher.init(Cipher.java:1143) at javax.crypto.Cipher.init(Cipher.java:1084) at com.android.org.conscrypt.CryptoUpcalls.rsaOpWithPrivateKey(CryptoUpcalls.java:173) at com.android.org.conscrypt.CryptoUpcalls.rsaSignDigestWithPrivateKey(CryptoUpcalls.java:132) at com.android.org.conscrypt.NativeCrypto.SSL_do_handshake(Native Method) at com.android.org.conscrypt.NativeSsl.doHandshake(NativeSsl.java:387) at com.android.org.conscrypt.ConscryptFileDescriptorSocket.startHandshake(ConscryptFileDescriptorSocket.java:226) at asynctls.mimecast.com.asynctlsapp.MainActivity$1.run(MainActivity.java:352) at java.lang.Thread.run(Thread.java:919)

W/CryptoUpcalls: Could not find provider for algorithm: RSA/ECB/NoPadding W/System.err: javax.net.ssl.SSLHandshakeException: Handshake failed W/System.err: at com.android.org.conscrypt.ConscryptFileDescriptorSocket.startHandshake(ConscryptFileDescriptorSocket.java:288) W/System.err: at asynctls.mimecast.com.asynctlsapp.MainActivity$1.run(MainActivity.java:351) W/System.err: at java.lang.Thread.run(Thread.java:919) W/System.err: Caused by: javax.net.ssl.SSLProtocolException: SSL handshake aborted: ssl=0x7dbcec0850c8: Failure in SSL library, usually a protocol error W/System.err: error:04000044:RSA routines:OPENSSL_internal:internal error (external/conscrypt/common/src/jni/main/cpp/conscrypt/native_crypto.cc:740 0x7dbce6155e73:0x00000000) W/System.err: at com.android.org.conscrypt.NativeCrypto.SSL_do_handshake(Native Method) W/System.err: at com.android.org.conscrypt.NativeSsl.doHandshake(NativeSsl.java:387) W/System.err: at com.android.org.conscrypt.ConscryptFileDescriptorSocket.startHandshake(ConscryptFileDescriptorSocket.java:226) W/System.err: … 2 more`

The app works fine with Pre Android 10 phone. Could you please look, this will block our app’s functionality completely.

Thanks,

[nildeka]

We are able to solve the problem. Adding setEncryptionPaddings(KeyProperties.ENCRYPTION_PADDING_NONE) during RSA Key-pair generation solved the problem.

kpg.initialize(new KeyGenParameterSpec.Builder( CLIENT_KEY_ALIAS, KeyProperties.PURPOSE_SIGN | KeyProperties.PURPOSE_VERIFY) .setDigests(KeyProperties.DIGEST_NONE, KeyProperties.DIGEST_SHA256, KeyProperties.DIGEST_SHA512) .setEncryptionPaddings(KeyProperties.ENCRYPTION_PADDING_NONE) .setSignaturePaddings(KeyProperties.SIGNATURE_PADDING_RSA_PKCS1) .build());

Looks like conscrypt has changed internal implementation of Signing handshake data. It worked in Pre-Android 10, because it only does Signing with private key. But from Android 10, it encrypts data with private key. In our case it encrypts with RSA/ECB/NoPadding. After adding encryption padding during key pair genration solved the problem.

Thanks

[satur9nine]

@farble1670 This appears to be the commit that broke us: https://github.com/google/conscrypt/commit/80469b4a9eae1f51d1c7af42963c452eb2d35fcd

Our custom keystore only allowed Signature and not Cipher so after this change conscrypt cannot use our keystore to perform mutual TLS. Ouch.