Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Unhandled exception in Paho `_thread_main` leaves the device client in a falsely 'connected' state

See original GitHub issue

Context

  • OS and version used: Debian GNU/Linux 11 (bullseye) (actually python:3.10-slim docker image)
  • Python version: Python 3.10.4
  • pip version: pip 22.0.4
  • list of installed packages:
Package             Version
------------------- ---------
azure-iot-device    2.11.0
certifi             2021.10.8
charset-normalizer  2.0.12
click               8.1.1
croniter            1.3.4
deprecation         2.1.0
idna                3.3
iotclient           0.6.0
janus               1.0.0
netifaces           0.11.0
packaging           21.3
paho-mqtt           1.6.1
peewee              3.14.10
pip                 22.0.4
pydantic            1.9.0
pyparsing           3.0.7
PySocks             1.7.1
python-dateutil     2.8.2
PyYAML              6.0
pyzmq               22.3.0
requests            2.27.1
requests-unixsocket 0.3.0
setuptools          58.1.0
six                 1.16.0
typing_extensions   4.1.1
urllib3             1.26.9
uvloop              0.16.0
wheel               0.37.1

Description of the issue

I have a device that was up and running for a couple of days until at one point where it permanently lost connectivity to IoT Hub. By digging through my logs I identified that this started happening after a thread, likely started by Paho, crashed due to an unhandled ssl.SSLEOFError. The device client did not seem to react to it and ended up behaving as if it was connected but not actually sending any messages through. This is the first time that I see this error, I would suspect it’s very rare. Connectivity did not recover until I manually restarted the application.

I see a maybe relevant/related issue in the Paho tracker here: https://github.com/eclipse/paho.mqtt.python/issues/637.

Code sample exhibiting the issue

I don’t know how to reproduce this explicitly or in what conditions this happens.

Console log of the issue

[2022-04-08 12:19:45,554] INFO:7:azure.iot.device.iothub.aio.async_clients: Sending method response to Hub...
[2022-04-08 12:19:45,556] INFO:7:azure.iot.device.common.mqtt_transport: publishing on $iothub/methods/res/200/?$rid=1
[2022-04-08 12:19:45,624] INFO:7:azure.iot.device.iothub.aio.async_clients: Sending method response to Hub...
[2022-04-08 12:19:45,624] INFO:7:azure.iot.device.common.mqtt_transport: disconnected with result code: 7
[2022-04-08 12:19:45,626] INFO:7:azure.iot.device.common.mqtt_transport: Forcing paho disconnect to prevent it from automatically reconnecting
[2022-04-08 12:19:45,626] INFO:7:azure.iot.device.common.mqtt_transport: publishing on $iothub/methods/res/200/?$rid=2
[2022-04-08 12:19:45,627] INFO:7:azure.iot.device.common.pipeline.pipeline_stages_mqtt: MQTTTransportStage: _on_mqtt_disconnect called: The connection was lost.
[2022-04-08 12:19:45,628] INFO:7:azure.iot.device.common.async_adapter: Callback completed with error The client is not currently connected.
[2022-04-08 12:19:45,629] INFO:7:azure.iot.device.common.async_adapter: ['azure.iot.device.common.transport_exceptions.NoConnectionError: The client is not currently connected.\n']
[2022-04-08 12:19:45,630] WARNING:7:IoTMessageGatewayMux: Ignoring raised unhandled exception while sending data through <AzureIoTHubConnectionAdapter object at 0x7f8cf9966b90>

<redacted>

Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/iotclient/services/connectivity/connection/iothub.py", line 204, in send_method_response
    await self._device_client.send_method_response(azure_response)
  File "/opt/venv/lib/python3.10/site-packages/azure/iot/device/aio/patch_documentation.py", line 66, in send_method_response
    return await super(IoTHubDeviceClient, self).send_method_response(method_response)
  File "/opt/venv/lib/python3.10/site-packages/azure/iot/device/iothub/aio/async_clients.py", line 418, in send_method_response
    await handle_result(callback)
  File "/opt/venv/lib/python3.10/site-packages/azure/iot/device/iothub/aio/async_clients.py", line 39, in handle_result
    raise exceptions.NoConnectionError("Client is not connected to IoTHub") from e
azure.iot.device.exceptions.NoConnectionError: Client is not connected to IoTHub
[2022-04-08 12:19:45,630] INFO:7:azure.iot.device.common.pipeline.pipeline_stages_mqtt: MQTTTransportStage: Unexpected disconnect (no pending connection op)
[2022-04-08 12:19:45,630] INFO:7:azure.iot.device.iothub.abstract_clients: Connection State - Disconnected
[2022-04-08 12:19:45,632] INFO:7:azure.iot.device.iothub.abstract_clients: Cleared all pending method requests due to disconnect
[2022-04-08 12:19:45,632] ERROR:7:azure.iot.device.common.handle_exceptions: Exception caught in background thread.  Unable to handle.
[2022-04-08 12:19:45,632] ERROR:7:azure.iot.device.common.handle_exceptions: ['azure.iot.device.common.transport_exceptions.ConnectionDroppedError: Unexpected disconnection\n']

[2022-04-08 12:19:45,634] INFO:7:AzureIoTHubConnectionAdapter: Connection lost!
[2022-04-08 12:19:45,634] INFO:7:AzureIoTHubConnectionAdapter: Attempting to connect to IoT Hub..
[2022-04-08 12:19:45,634] INFO:7:azure.iot.device.iothub.aio.async_clients: Connecting to Hub...
[2022-04-08 12:19:45,636] INFO:7:azure.iot.device.common.mqtt_transport: Connect using port 8883 (TCP)
[2022-04-08 12:19:49,552] INFO:7:azure.iot.device.common.mqtt_transport: connected with result code: 0
[2022-04-08 12:19:49,555] INFO:7:azure.iot.device.common.pipeline.pipeline_stages_mqtt: _on_mqtt_connected called
[2022-04-08 12:19:49,557] INFO:7:azure.iot.device.iothub.abstract_clients: Connection State - Connected
[2022-04-08 12:19:49,564] INFO:7:azure.iot.device.iothub.aio.async_clients: Successfully connected to Hub
[2022-04-08 12:19:49,564] INFO:7:AzureIoTHubConnectionAdapter: Connection established!

[2022-04-08 12:19:49,569] INFO:7:azure.iot.device.iothub.aio.async_clients: Sending message to Hub...
[2022-04-08 12:19:49,570] INFO:7:azure.iot.device.iothub.aio.async_clients: Sending message to Hub...
[2022-04-08 12:19:49,571] INFO:7:azure.iot.device.common.mqtt_transport: publishing on <redacted>
[2022-04-08 12:19:49,571] INFO:7:azure.iot.device.common.mqtt_transport: message received on $iothub/methods/POST/<redacted>/?$rid=1
[2022-04-08 12:19:49,573] INFO:7:azure.iot.device.common.mqtt_transport: publishing on <redacted>
[2022-04-08 12:19:49,583] INFO:7:azure.iot.device.iothub.aio.async_clients: Received method request
[2022-04-08 12:19:49,587] INFO:7:azure.iot.device.iothub.aio.async_clients: Waiting for method request...
[2022-04-08 12:19:49,593] INFO:7:azure.iot.device.common.mqtt_transport: message received on $iothub/methods/POST/<redacted>/?$rid=2
Exception in thread Thread-17221 (_thread_main):
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/threading.py", line 1009, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.10/threading.py", line 946, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/venv/lib/python3.10/site-packages/paho/mqtt/client.py", line 3591, in _thread_main
    self.loop_forever(retry_first_connection=True)
  File "/opt/venv/lib/python3.10/site-packages/paho/mqtt/client.py", line 1756, in loop_forever
    rc = self._loop(timeout)
  File "/opt/venv/lib/python3.10/site-packages/paho/mqtt/client.py", line 1181, in _loop
    rc = self.loop_write()
  File "/opt/venv/lib/python3.10/site-packages/paho/mqtt/client.py", line 1577, in loop_write
    rc = self._packet_write()
  File "/opt/venv/lib/python3.10/site-packages/paho/mqtt/client.py", line 2464, in _packet_write
    write_length = self._sock_send(
  File "/opt/venv/lib/python3.10/site-packages/paho/mqtt/client.py", line 649, in _sock_send
    return self._sock.send(buf)
  File "/usr/local/lib/python3.10/ssl.py", line 1205, in send
    return self._sslobj.write(data)
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:2396)
[2022-04-08 12:19:49,709] INFO:7:azure.iot.device.iothub.aio.async_clients: Received method request
[2022-04-08 12:19:49,712] INFO:7:azure.iot.device.iothub.aio.async_clients: Waiting for method request...
[2022-04-08 12:19:51,504] INFO:7:azure.iot.device.iothub.aio.async_clients: Sending method response to Hub...
[2022-04-08 12:19:51,507] INFO:7:azure.iot.device.common.mqtt_transport: publishing on $iothub/methods/res/200/?$rid=1
[2022-04-08 12:19:51,591] INFO:7:azure.iot.device.iothub.aio.async_clients: Sending method response to Hub...
[2022-04-08 12:19:51,593] INFO:7:azure.iot.device.common.mqtt_transport: publishing on $iothub/methods/res/200/?$rid=2

<redacted, nothing interesting>

[2022-04-08 12:20:14,721] INFO:7:azure.iot.device.iothub.aio.async_clients: Sending message to Hub...
[2022-04-08 12:20:14,723] INFO:7:azure.iot.device.common.mqtt_transport: publishing on <redacted>
[2022-04-08 12:20:19,572] INFO:7:azure.iot.device.iothub.aio.async_clients: Sending message to Hub...
[2022-04-08 12:20:19,573] INFO:7:azure.iot.device.common.mqtt_transport: publishing on <redacted>
[2022-04-08 12:20:44,745] INFO:7:azure.iot.device.iothub.aio.async_clients: Sending message to Hub...
[2022-04-08 12:20:44,746] INFO:7:azure.iot.device.common.mqtt_transport: publishing on <redacted>

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:8 (5 by maintainers)

github_iconTop GitHub Comments

1reaction
cartertinneycommented, Apr 19, 2022

@zoopp

Thanks for bringing this to our attention. I suspect you are correct and that this is a bug in the Paho library. We’ll look into it.

0reactions
cartertinneycommented, Jun 27, 2022

Closing due to inactivity. We are monitoring the Paho GitHub issue however.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Cannot get Python Device SDK to connect "Error in the ...
Cannot get Python Device SDK to connect "Error in the IoTHub client due to TLS exchanges." Hello, I have written some code to...
Read more >
Connecting with M2MQTT Paho Python client to Azure IoT Hub
I am getting now: "Failed to connect.Connection refused, server unavailable. Error Code= 3". If your code really works the only difference I see ......
Read more >
Paho Python MQTT Client-Understanding The Loop
If your client script has more than one client connection then you must call or start a loop for each client connection.
Read more >
Publish MQTT messages to AWS IoT Core using Python
I can't send or receive MQTT (MQ Telemetry Transport) messages between AWS IoT Core and my device or MQTT client. How do I...
Read more >
if !() statement in twig Code Example - Code Grepper
{% if online == false %}. 2. <p>Our website is in maintenance mode. Please, come back later.</p>. 3. {% endif %}. Source: twig.symfony.com....
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Connecting to EdgeHub fails after several successful messages

Next page

[Edge Transparent Gateway scenario]Send message using x509=true in connection string