Azure Event Hub resets connection

Hi all,

Microsoft Azure offers a service they call Azure Event Hub, which can collect events from Kafka producers

Here’s an example from Microsoft using the confluent_kafka module: https://github.com/Azure/azure-event-hubs/blob/master/samples/kafka/python/producer.py

However, when I try similar steps with kafka-python, Azure disconnects. Any ideas?

import ssl

from kafka import KafkaProducer

class KafkaError(RuntimeError):
    """Raised when a Kafka error occurs"""

class KafkaClient(object):
    def __init__(self, kafka_hosts, use_ssl=False, username=None,
                 password=None):
        """
        Initializes the Kafka client
        Args:
            kafka_hosts (list): A list of Kafka hostnames
            (with optional port numbers)
            use_ssl (bool): Use a SSL/TLS connection
            username (str): An optional username
            password (str):  An optional password

        Notes:
            ``use_ssl=True`` is implied when a username or password are
            supplied.

            When using Azure Event Hubs, the username is literally
            ``$ConnectionString``, and the password is the
            Azure Event Hub connection string.
        """
        config = dict(value_serializer=lambda v: json.dumps(v).encode(
                              'utf-8'),
                      bootstrap_servers=kafka_hosts)
        if use_ssl or username or password:
            config["security_protocol"] = "SSL"
            config["ssl_context"] = ssl.create_default_context()
            if username or password:
                config["sasl_plain_username"] = username or ""
                config["sasl_plain_password"] = password or ""
        try:
            self.producer = KafkaProducer(**config)
        except NoBrokersAvailable:
            raise KafkaError("No Kafka brokers available")

ERROR:kafka.conn:<BrokerConnection node_id=bootstrap host=parsedmarc.servicebus.windows.net:9093 <connected> [IPv4 ('40.71.10.128', 9093)]>: Error receiving network data closing socket
Traceback (most recent call last):
  File "C:\Users\Sean\GitHub\parsedmarc\venv\lib\site-packages\kafka\conn.py", line 816, in _recv
    data = self._sock.recv(self.config['sock_chunk_bytes'])
  File "C:\Program Files\Python37\lib\ssl.py", line 1034, in recv
    return self.read(buflen)
  File "C:\Program Files\Python37\lib\ssl.py", line 910, in read
    return self._sslobj.read(len)
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host
ERROR:kafka.client:Unable to bootstrap from [('parsedmarc.servicebus.windows.net', 9093, <AddressFamily.AF_UNSPEC: 0>)]

[arerlend]

@seanthegeek in version 1.4.4, this configuration works for me -

connection_string = 'Endpoint=sb://[namespace].servicebus.windows.net/;SharedAccessKeyName=[key name];SharedAccessKey=[key]
servers = '[namespace].servicebus.windows.net:9093'

producer = KafkaProducer(
    bootstrap_servers=servers,
    security_protocol='SASL_SSL', # this is likely the problem
    sasl_mechanism='PLAIN',
    sasl_plain_username='$ConnectionString',
    sasl_plain_password=connection_string,
    ssl_cafile=ca_path,
    api_version=(2,0,0) # for example - this should work for (1,0,0) and greater
)

In your case, we may be forcibly closing the connection because we’re expecting you to use the SSL plugin via SASL (API key 17 - first the handshake, then the auth bytes).

[Idofriedman]

ssl_cafile=ca_path,

What is the certificate? Where do I get it from?