Can't do just in time provisioning at AWS IoT, certificate contains CA

Describe the bug

I’m trying to do just in time provisioning at AWS IoT by following this article https://aws.amazon.com/blogs/iot/setting-up-just-in-time-provisioning-with-aws-iot-core/
I can do it with external tools like Mqtt explorer but can’t with MQTTNet package.

Which project is your bug related to?

• MQTTnet nuget package 3.0.13

To Reproduce

Steps to reproduce the behavior:

1. Having AWS IoT instance
2. Create and register own CA by following steps from here - https://aws.amazon.com/blogs/iot/setting-up-just-in-time-provisioning-with-aws-iot-core/
3. Create private key and certificate issued from the CA earlier registered at AWS IoT by using openssl
4. REMARK. By using mqtt explorer I can do provisioning, just need to send CA together with certificate. To accomplish that need to create a file that contains certificate and CA. Use this bash command: cat cert.pem ca.pem > certAndCA.pem. Use this file as a certificate to establish mqtt connection. Everything works fine.
5. To to the same in .net code need to create pfx certificate that contains certificate, CA and private key. Use this command: openssl pkcs12 -export -out certAndCA.pfx -inkey keyfile.pem -in cert.pem -certfile ca.pem
6. Try to connect. App fails with a timeout. Trace logs contains next warning Timeout while waiting for packet of type 'MqttConnAckPacket'

Expected behavior

I guess it should work. Not sure where the issue is. I tried different approaches. For example, created pfx certificate that contains only certificate and private key and added separately ca certificate. No luck

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context / logging

Add any other context about the problem here. Include debugging or logging information here:

>> [2021-02-01T20:54:00.0348183Z] [1] [MqttClient] [Verbose]: Trying to connect with server '{aws_iot}.amazonaws.com:8883' (Timeout=00:00:10).
>> [2021-02-01T20:54:00.8481910Z] [7] [MqttClient] [Verbose]: Connection with server established.
>> [2021-02-01T20:54:00.8738003Z] [5] [MqttClient] [Verbose]: Start receiving packets.
>> [2021-02-01T20:54:00.9222467Z] [7] [MqttChannelAdapter] [Verbose]: TX (26 bytes) >>> Connect: [ClientId=uniqueclient] [Username=] [Password=] [KeepAlivePeriod=60] [CleanSession=True]
>> [2021-02-01T20:54:00.9987905Z] [7] [MqttClient] [Verbose]: Disconnecting [Timeout=00:00:10]
>> [2021-02-01T20:54:01.0036639Z] [7] [MqttClient] [Verbose]: Disconnected from adapter.
>> [2021-02-01T20:54:01.0073583Z] [7] [MqttClient] [Info]: Disconnected.
>> [2021-02-01T20:54:01.0077173Z] [7] [MqttClient] [Verbose]: Stopped receiving packets.
>> [2021-02-01T20:54:10.9412866Z] [4] [MqttClient] [Warning]: Timeout while waiting for packet of type 'MqttConnAckPacket'.
>> [2021-02-01T20:54:10.9473162Z] [4] [MqttClient] [Error]: Error while connecting with server.
MQTTnet.Exceptions.MqttCommunicationTimedOutException: Exception of type 'MQTTnet.Exceptions.MqttCommunicationTimedOutException' was thrown.
   at MQTTnet.PacketDispatcher.MqttPacketAwaiter`1.WaitOneAsync(TimeSpan timeout)
   at MQTTnet.Client.MqttClient.SendAndReceiveAsync[TResponsePacket](MqttBasePacket requestPacket, CancellationToken cancellationToken)
   at MQTTnet.Client.MqttClient.AuthenticateAsync(IMqttChannelAdapter channelAdapter, MqttApplicationMessage willApplicationMessage, CancellationToken cancellationToken)
   at MQTTnet.Client.MqttClient.ConnectAsync(IMqttClientOptions options, CancellationToken cancellationToken)

Wireshark logs shows that server sent and Encrypted Alert

68	8.316424	192.168.0.103	13.48.179.113	TCP	66	1506 → 8883 [SYN] Seq=0 Win=65535 Len=0 MSS=1460 WS=2 SACK_PERM=1
69	8.362340	13.48.179.113	192.168.0.103	TCP	66	8883 → 1506 [SYN, ACK] Seq=0 Ack=1 Win=26883 Len=0 MSS=1460 SACK_PERM=1 WS=256
70	8.362634	192.168.0.103	13.48.179.113	TCP	54	1506 → 8883 [ACK] Seq=1 Ack=1 Win=65536 Len=0
71	8.477410	192.168.0.103	13.48.179.113	TLSv1.2	263	Client Hello
73	8.523059	13.48.179.113	192.168.0.103	TCP	54	8883 → 1506 [ACK] Seq=1 Ack=210 Win=28160 Len=0
74	8.523588	13.48.179.113	192.168.0.103	TLSv1.2	150	Server Hello
75	8.523708	13.48.179.113	192.168.0.103	TCP	1514	8883 → 1506 [ACK] Seq=97 Ack=210 Win=28160 Len=1460 [TCP segment of a reassembled PDU]
76	8.523759	192.168.0.103	13.48.179.113	TCP	54	1506 → 8883 [ACK] Seq=210 Ack=1557 Win=65536 Len=0
77	8.524372	13.48.179.113	192.168.0.103	TCP	2974	8883 → 1506 [ACK] Seq=1557 Ack=210 Win=28160 Len=2920 [TCP segment of a reassembled PDU]
78	8.524372	13.48.179.113	192.168.0.103	TLSv1.2	565	Certificate
79	8.524435	192.168.0.103	13.48.179.113	TCP	54	1506 → 8883 [ACK] Seq=210 Ack=4988 Win=65536 Len=0
80	8.524500	13.48.179.113	192.168.0.103	TLSv1.2	438	Server Key Exchange, Certificate Request, Server Hello Done
81	8.524523	192.168.0.103	13.48.179.113	TCP	54	1506 → 8883 [ACK] Seq=210 Ack=5372 Win=65152 Len=0
83	8.571575	192.168.0.103	13.48.179.113	TLSv1.2	1309	Certificate, Client Key Exchange, Certificate Verify, Change Cipher Spec, Encrypted Handshake Message
84	8.618627	13.48.179.113	192.168.0.103	TLSv1.2	105	Change Cipher Spec, Encrypted Handshake Message
85	8.659562	192.168.0.103	13.48.179.113	TCP	54	1506 → 8883 [ACK] Seq=1465 Ack=5423 Win=65100 Len=0
86	8.721786	192.168.0.103	13.48.179.113	TLSv1.2	109	Application Data
87	8.777379	13.48.179.113	192.168.0.103	TLSv1.2	85	Encrypted Alert
88	8.796676	192.168.0.103	13.48.179.113	TCP	54	1506 → 8883 [FIN, ACK] Seq=1520 Ack=5454 Win=65070 Len=0
89	8.843021	13.48.179.113	192.168.0.103	TCP	54	8883 → 1506 [FIN, ACK] Seq=5454 Ack=1521 Win=30720 Len=0
90	8.843313	192.168.0.103	13.48.179.113	TCP	54	1506 → 8883 [ACK] Seq=1521 Ack=5455 Win=65070 Len=0

Code example

Please provide full code examples below where possible to make it easier for the developers to check your issues.

using System;
using System.Security.Authentication;
using System.Security.Cryptography.X509Certificates;
using System.Threading.Tasks;
using MQTTnet.Client;
using MQTTnet.Client.Options;
using MQTTnet.Diagnostics;

namespace MQTTnet.Spike
{
    class Program
    {
        static async Task Main(string[] args)
        {
            TraceInternalLogsToConsole();
            IMqttClientOptions options = CreateOptions();
            IMqttClient client = new MqttFactory()
                .CreateMqttClient();

            await client.ConnectAsync(options);

            Console.WriteLine("Connected!");
            Console.WriteLine("Press any key to exit");
            Console.ReadKey();
        }

        private static IMqttClientOptions CreateOptions()
        {
            var certAndCaPfx = new X509Certificate2(
                @"..\certAndCA.pfx",
                string.Empty,
                X509KeyStorageFlags.PersistKeySet | X509KeyStorageFlags.Exportable);
            var result = new MqttClientOptionsBuilder()
                .WithCleanSession()
                .WithClientId("anyId")
                .WithKeepAlivePeriod(TimeSpan.FromSeconds(60))
                .WithTcpServer("{aws_iot}.amazonaws.com", 8883)
                .WithTls(new MqttClientOptionsBuilderTlsParameters
                {
                    UseTls = true,
                    SslProtocol = SslProtocols.Tls12,
                    Certificates = new[] { certAndCaPfx },
                    IgnoreCertificateChainErrors = true,
                    IgnoreCertificateRevocationErrors = true,
                    AllowUntrustedCertificates = true,
                    CertificateValidationHandler = context => true
                }).Build();
            return result;

        }

        private static void TraceInternalLogsToConsole()
        {
            MqttNetGlobalLogger.LogMessagePublished += (s, e) =>
            {
                var trace = $">> [{e.TraceMessage.Timestamp:O}] [{e.TraceMessage.ThreadId}] [{e.TraceMessage.Source}] [{e.TraceMessage.Level}]: {e.TraceMessage.Message}";
                if (e.TraceMessage.Exception != null)
                {
                    trace += Environment.NewLine + e.TraceMessage.Exception.ToString();
                }
                Console.WriteLine(trace);
            };
        }
    }
}

[SeppPenner]

I have added your troubleshooting tips to https://github.com/chkr1011/MQTTnet/wiki/Client#troubleshooting, @albert-t25 😉

[albert-t25]

Thank you @KirillSelyak . My thinking is to try and pass the intermediate certificate as part of the options

                var tlsParameters = new MqttClientOptionsBuilderTlsParameters
                {
                    UseTls = true,
                    Certificates = new[] { Certificate, IntermediateCa, RootCertificate },
                    IgnoreCertificateRevocationErrors = false, //Prevent online revocation check
                    IgnoreCertificateChainErrors = false,
                    AllowUntrustedCertificates = false,
                    SslProtocol = System.Security.Authentication.SslProtocols.Tls12
                };

I’m not sure if it will work, but seems worth a try.