Keep getting ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1076)

Hello, I am using kafka-python>=2.0.1 (with Python 3.7.6 on MacOsx, build from the python website) it generally works fine, but when I tried to access our broker, that has SSL auth, I will get the above error. it happens if I enable SSL with my certificates. I’m sure that there’s one that is self signed internally in my company.

return KafkaProducer(
            bootstrap_servers=self.bootstrap_servers,
            acks='all',
            compression_type=None,
            retries=5,
            batch_size=16384 * 5,
            max_block_ms=5000,
            retry_backoff_ms=100 * 10,
            linger_ms=5,
            client_id='data-importer',
            security_protocol='SSL',
            ssl_check_hostname=True,
            api_version=(0, 20),
            ssl_cafile=rel_to(__file__, '../kafkakeys/KafkaClientCAChain.pem'),
            ssl_certfile=rel_to(__file__, '../kafkakeys/certificate.pem'),
            ssl_keyfile=rel_to(__file__, '../kafkakeys/key.pem'),

with this, when sending a message, it gets stuck in a loop and prints:

Traceback (most recent call last):
  File "/Users/-----/dev/prj/data-importer-python/.venv/lib/python3.7/site-packages/kafka/producer/sender.py", line 60, in run
    self.run_once()
  File "/Users/-----/dev/prj/data-importer-python/.venv/lib/python3.7/site-packages/kafka/producer/sender.py", line 160, in run_once
    self._client.poll(timeout_ms=poll_timeout_ms)
  File "/Users/-----/dev/prj/data-importer-python/.venv/lib/python3.7/site-packages/kafka/client_async.py", line 600, in poll
    self._poll(timeout / 1000)
  File "/Users/d-----i/dev/prj/data-importer-python/.venv/lib/python3.7/site-packages/kafka/client_async.py", line 646, in _poll
    conn.connect()
  File "/Users/d-----i/dev/prj/data-importer-python/.venv/lib/python3.7/site-packages/kafka/conn.py", line 426, in connect
    if self._try_handshake():
  File "/Users/-----i/dev/prj/data-importer-python/.venv/lib/python3.7/site-packages/kafka/conn.py", line 505, in _try_handshake
    self._sock.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 1139, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1076)
ERROR:kafka.producer.sender:Uncaught error in kafka producer I/O thread
Traceback (most recent call last):
  File "/Users/-----/dev/prj/data-importer-python/.venv/lib/python3.7/site-packages/kafka/producer/sender.py", line 60, in run
    self.run_once()
  File "/Users/-----dev/prj/data-importer-python/.venv/lib/python3.7/site-packages/kafka/producer/sender.py", line 160, in run_once
    self._client.poll(timeout_ms=poll_timeout_ms)
  File "/Users/-----/dev/prj/data-importer-python/.venv/lib/python3.7/site-packages/kafka/client_async.py", line 580, in poll
    self._maybe_connect(node_id)
  File "/Users/=------/dev/prj/data-importer-python/.venv/lib/python3.7/site-packages/kafka/client_async.py", line 390, in _maybe_connect
    conn.connect()
  File "/Users/-----/dev/prj/data-importer-python/.venv/lib/python3.7/site-packages/kafka/conn.py", line 426, in connect
    if self._try_handshake():
  File "/Users/-----/dev/prj/data-importer-python/.venv/lib/python3.7/site-packages/kafka/conn.py", line 505, in _try_handshake
    self._sock.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 1139, in do_handshake
    self._sslobj.do_handshake()
OSError: [Errno 0] Error

to solve this I tried a number of python installations (provided by brew, pyenv and eventually the installer from the python website). I tried to fix the issue by running Install Certificates.command that reinstalls the certificates. but it keeps doing so.

If I remove ssl_cafile and ssl_certfile (or just one of the two, leaving ssl_keyfile) it will stop giving that exception, but will just die with

kafka.errors.KafkaTimeoutError: KafkaTimeoutError: Failed to update metadata after 5.0 secs.

Can anyone help please?

[abiodunjames]

I’m currently using the sslcontext as a workaround

cert = "user.crt"
key = "user.key"
context = ssl.create_default_context()
context.load_cert_chain(certfile=cert, keyfile=key)
context.check_hostname = False
context.verify_mode = ssl.CERT_NONE
self.ssl_context = context
self.security_protocol = "SSL"

[dechoma]

it is significant where you get your ssl_cafile from

when i was using ca.crt ( from user’s secret ) as ssl_cafile then I get an error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain , when I switched to the ca.crt from secret cluster-ca-cert it worked like a charm