Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Unusual/poor connection to IoT Hub

See original GitHub issue

I’m not sure if this is related to #812 or #810 but it seems to me that the client gets stuck in a state where it can’t send any messages. I’m not completely certain as at this stage it could still be other things - and it’s also odd as some devices seen fine, and others not.

I’m basically following this example, and

On Raspberry Pi 3A+ with Raspbian
Over a low bandwidth (15kb) cellular connection
Python 3.7
and

$ pip freeze | grep azure
azure-core==1.16.0
azure-iot-device==2.7.1
azure-iot-hub==2.4.0

Firstly

Aug 12 07:56:29 *** python[508]: ReconnectStage: DisconnectEvent received while in unexpected state - WAITING_TO_RECONNECT, Connected: False

Which seems odd because of here, and the comment # BAD STATE (this block should not be reached) but I obviously have reached it. When I run without changing the loglevel, I just see those repeating and no other warnings etc. When I enabled debug logging, I’m seeing the below - unfortunately it doesn’t include the above message (as getting logs is difficult) but it may point to an error. I will try to get more complete logs if possible.

-- Logs begin at Thu 2021-08-12 07:39:45 NZST, end at Thu 2021-08-12 08:05:14 NZST. --
Aug 12 08:02:57 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_thread:Starting on_reconnect_timer_expired in pipeline thread
Aug 12 08:02:57 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:ReconnectStage: Reconnect timer expired. State is WAITING_TO_RECONNECT Connected is False.
Aug 12 08:02:57 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:ReconnectStage: sending new connect op down
Aug 12 08:02:57 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:ConnectionLockStage(ConnectOperation): blocking
Aug 12 08:02:57 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_mqtt:MQTTTransportStage(ConnectOperation): connecting
Aug 12 08:02:57 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_mqtt:MQTTTransportStage(ConnectOperation): Starting watchdog
Aug 12 08:02:57 *** python[17028]: DEBUG:azure.iot.device.common.mqtt_transport:connecting to mqtt broker
Aug 12 08:02:57 *** python[17028]: INFO:azure.iot.device.common.mqtt_transport:Connect using port 8883 (TCP)
Aug 12 08:03:02 *** python[17028]: DEBUG:paho:Sending CONNECT (u1, p1, wr0, wq0, wf0, c0, k60) client_id=b'***'
Aug 12 08:03:02 *** python[17028]: DEBUG:azure.iot.device.common.mqtt_transport:_mqtt_client.connect returned rc=0
Aug 12 08:03:03 *** python[17028]: DEBUG:paho:Received CONNACK (1, 0)
Aug 12 08:03:03 *** python[17028]: INFO:azure.iot.device.common.mqtt_transport:connected with result code: 0
Aug 12 08:03:03 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_thread:Starting _on_mqtt_connected in pipeline thread
Aug 12 08:03:03 *** python[17028]: INFO:azure.iot.device.common.pipeline.pipeline_stages_mqtt:_on_mqtt_connected called
Aug 12 08:03:03 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:PipelineRootStage: ConnectedEvent received. Calling on_connected_handler
Aug 12 08:03:03 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_thread:Starting _on_connected in callback thread
Aug 12 08:03:03 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_mqtt:completing connect op
Aug 12 08:03:03 *** python[17028]: INFO:azure.iot.device.iothub.abstract_clients:Connection State - Connected
Aug 12 08:03:03 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_mqtt:MQTTTransportStage(ConnectOperation): cancelling watchdog
Aug 12 08:03:03 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_ops_base:ConnectOperation: completing without error
Aug 12 08:03:03 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:ConnectionLockStage(ConnectOperation): op succeeded.  Unblocking queue
Aug 12 08:03:03 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:ConnectionLockStage(ConnectOperation): unblocking and releasing queued ops.
Aug 12 08:03:03 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:ConnectionLockStage(ConnectOperation): processing 0 items in queue for error=None
Aug 12 08:03:03 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:ReconnectStage(ConnectOperation): on_connect_complete error=None state=LOGICALLY_CONNECTED never_connected=False connected=True
Aug 12 08:03:03 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:ReconnectStage: completing waiting ops with error=None
Aug 12 08:03:04 *** python[17028]: INFO:azure.iot.device.common.mqtt_transport:disconnected with result code: 1
Aug 12 08:03:04 *** python[17028]: DEBUG:azure.iot.device.common.mqtt_transport:  File "/usr/lib/python3.7/threading.py", line 885, in _bootstrap
Aug 12 08:03:04 *** python[17028]:     self._bootstrap_inner()
Aug 12 08:03:04 *** python[17028]:   File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
Aug 12 08:03:04 *** python[17028]:     self.run()
Aug 12 08:03:04 *** python[17028]:   File "/usr/lib/python3.7/threading.py", line 865, in run
Aug 12 08:03:04 *** python[17028]:     self._target(*self._args, **self._kwargs)
Aug 12 08:03:04 *** python[17028]:   File "/usr/local/lib/python3.7/dist-packages/paho/mqtt/client.py", line 3452, in _thread_main
Aug 12 08:03:04 *** python[17028]:     self.loop_forever(retry_first_connection=True)
Aug 12 08:03:04 *** python[17028]:   File "/usr/local/lib/python3.7/dist-packages/paho/mqtt/client.py", line 1779, in loop_forever
Aug 12 08:03:04 *** python[17028]:     rc = self.loop(timeout, max_packets)
Aug 12 08:03:04 *** python[17028]:   File "/usr/local/lib/python3.7/dist-packages/paho/mqtt/client.py", line 1181, in loop
Aug 12 08:03:04 *** python[17028]:     rc = self.loop_read(max_packets)
Aug 12 08:03:04 *** python[17028]:   File "/usr/local/lib/python3.7/dist-packages/paho/mqtt/client.py", line 1574, in loop_read
Aug 12 08:03:04 *** python[17028]:     return self._loop_rc_handle(rc)
Aug 12 08:03:04 *** python[17028]:   File "/usr/local/lib/python3.7/dist-packages/paho/mqtt/client.py", line 2227, in _loop_rc_handle
Aug 12 08:03:04 *** python[17028]:     self._do_on_disconnect(rc, properties)
Aug 12 08:03:04 *** python[17028]:   File "/usr/local/lib/python3.7/dist-packages/paho/mqtt/client.py", line 3360, in _do_on_disconnect
Aug 12 08:03:04 *** python[17028]:     self.on_disconnect(self, self._userdata, rc)
Aug 12 08:03:04 *** python[17028]:   File "/usr/local/lib/python3.7/dist-packages/azure/iot/device/common/mqtt_transport.py", line 210, in on_disconnect
Aug 12 08:03:04 *** python[17028]:     logger.debug("".join(traceback.format_stack()))
Aug 12 08:03:04 *** python[17028]: INFO:azure.iot.device.common.mqtt_transport:Forcing paho disconnect to prevent it from automatically reconnecting
Aug 12 08:03:04 *** python[17028]: DEBUG:azure.iot.device.common.mqtt_transport:in paho thread.  nulling _thread
Aug 12 08:03:04 *** python[17028]: DEBUG:azure.iot.device.common.mqtt_transport:Done forcing paho disconnect
Aug 12 08:03:04 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_thread:Starting _on_mqtt_disconnected in pipeline thread
Aug 12 08:03:04 *** python[17028]: INFO:azure.iot.device.common.pipeline.pipeline_stages_mqtt:MQTTTransportStage: _on_mqtt_disconnect called: ConnectionDroppedError('Paho returned rc==1')
Aug 12 08:03:04 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:ReconnectStage(DisconnectedEvent): State is LOGICALLY_CONNECTED Connected is True.
Aug 12 08:03:04 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:ReconnectStage(DisconnectedEvent): Attempting to reconnect
Aug 12 08:03:04 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:PipelineRootStage: DisconnectedEvent received. Calling on_disconnected_handler
Aug 12 08:03:04 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_thread:Starting _on_disconnected in callback thread
Aug 12 08:03:04 *** python[17028]: INFO:azure.iot.device.common.pipeline.pipeline_stages_mqtt:MQTTTransportStage: disconnection was unexpected
Aug 12 08:03:04 *** python[17028]: INFO:azure.iot.device.iothub.abstract_clients:Connection State - Disconnected
Aug 12 08:03:04 *** python[17028]: INFO:azure.iot.device.iothub.abstract_clients:Cleared all pending method requests due to disconnect
Aug 12 08:03:04 *** python[17028]: INFO:azure.iot.device.common.handle_exceptions:Unexpected disconnection
Aug 12 08:03:04 *** python[17028]: INFO:azure.iot.device.common.handle_exceptions:azure.iot.device.common.transport_exceptions.ConnectionDroppedError: ConnectionDroppedError('Paho returned rc==1')
Aug 12 08:03:04 *** python[17028]: The above exception was the direct cause of the following exception:
Aug 12 08:03:04 *** python[17028]: Traceback (most recent call last):
Aug 12 08:03:04 *** python[17028]:   File "/usr/local/lib/python3.7/dist-packages/azure/iot/device/common/handle_exceptions.py", line 43, in swallow_unraised_exception
Aug 12 08:03:04 *** python[17028]:     raise e
Aug 12 08:03:04 *** python[17028]: azure.iot.device.common.transport_exceptions.ConnectionDroppedError: ConnectionDroppedError(None) caused by ConnectionDroppedError('Paho returned rc==1')
Aug 12 08:03:04 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_thread:Starting on_reconnect_timer_expired in pipeline thread
Aug 12 08:03:04 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:ReconnectStage: Reconnect timer expired. State is WAITING_TO_RECONNECT Connected is False.
Aug 12 08:03:04 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:ReconnectStage: sending new connect op down
Aug 12 08:03:04 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:ConnectionLockStage(ConnectOperation): blocking
Aug 12 08:03:04 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_mqtt:MQTTTransportStage(ConnectOperation): connecting
Aug 12 08:03:04 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_mqtt:MQTTTransportStage(ConnectOperation): Starting watchdog
Aug 12 08:03:04 *** python[17028]: DEBUG:azure.iot.device.common.mqtt_transport:connecting to mqtt broker
Aug 12 08:03:04 *** python[17028]: INFO:azure.iot.device.common.mqtt_transport:Connect using port 8883 (TCP)
Aug 12 08:03:10 *** python[17028]: DEBUG:paho:Sending CONNECT (u1, p1, wr0, wq0, wf0, c0, k60) client_id=b'***'
Aug 12 08:03:10 *** python[17028]: DEBUG:azure.iot.device.common.mqtt_transport:_mqtt_client.connect returned rc=0
Aug 12 08:03:10 *** python[17028]: DEBUG:paho:Received CONNACK (1, 0)
Aug 12 08:03:10 *** python[17028]: INFO:azure.iot.device.common.mqtt_transport:connected with result code: 0
Aug 12 08:03:10 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_thread:Starting _on_mqtt_connected in pipeline thread
Aug 12 08:03:10 *** python[17028]: INFO:azure.iot.device.common.pipeline.pipeline_stages_mqtt:_on_mqtt_connected called
Aug 12 08:03:10 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:PipelineRootStage: ConnectedEvent received. Calling on_connected_handler
Aug 12 08:03:10 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_thread:Starting _on_connected in callback thread
Aug 12 08:03:10 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_mqtt:completing connect op
Aug 12 08:03:10 *** python[17028]: INFO:azure.iot.device.iothub.abstract_clients:Connection State - Connected
Aug 12 08:03:10 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_mqtt:MQTTTransportStage(ConnectOperation): cancelling watchdog
Aug 12 08:03:10 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_ops_base:ConnectOperation: completing without error
Aug 12 08:03:10 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:ConnectionLockStage(ConnectOperation): op succeeded.  Unblocking queue
Aug 12 08:03:10 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:ConnectionLockStage(ConnectOperation): unblocking and releasing queued ops.
Aug 12 08:03:10 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:ConnectionLockStage(ConnectOperation): processing 0 items in queue for error=None
Aug 12 08:03:10 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:ReconnectStage(ConnectOperation): on_connect_complete error=None state=LOGICALLY_CONNECTED never_connected=False connected=True
Aug 12 08:03:10 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:ReconnectStage: completing waiting ops with error=None
Aug 12 08:03:14 *** python[17028]: INFO:azure.iot.device.common.mqtt_transport:disconnected with result code: 1
Aug 12 08:03:14 *** python[17028]: DEBUG:azure.iot.device.common.mqtt_transport:  File "/usr/lib/python3.7/threading.py", line 885, in _bootstrap
Aug 12 08:03:14 *** python[17028]:     self._bootstrap_inner()
Aug 12 08:03:14 *** python[17028]:   File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
Aug 12 08:03:14 *** python[17028]:     self.run()
Aug 12 08:03:14 *** python[17028]:   File "/usr/lib/python3.7/threading.py", line 865, in run
Aug 12 08:03:14 *** python[17028]:     self._target(*self._args, **self._kwargs)
Aug 12 08:03:14 *** python[17028]:   File "/usr/local/lib/python3.7/dist-packages/paho/mqtt/client.py", line 3452, in _thread_main
Aug 12 08:03:14 *** python[17028]:     self.loop_forever(retry_first_connection=True)
Aug 12 08:03:14 *** python[17028]:   File "/usr/local/lib/python3.7/dist-packages/paho/mqtt/client.py", line 1779, in loop_forever
Aug 12 08:03:14 *** python[17028]:     rc = self.loop(timeout, max_packets)
Aug 12 08:03:14 *** python[17028]:   File "/usr/local/lib/python3.7/dist-packages/paho/mqtt/client.py", line 1181, in loop
Aug 12 08:03:14 *** python[17028]:     rc = self.loop_read(max_packets)
Aug 12 08:03:14 *** python[17028]:   File "/usr/local/lib/python3.7/dist-packages/paho/mqtt/client.py", line 1574, in loop_read
Aug 12 08:03:14 *** python[17028]:     return self._loop_rc_handle(rc)
Aug 12 08:03:14 *** python[17028]:   File "/usr/local/lib/python3.7/dist-packages/paho/mqtt/client.py", line 2227, in _loop_rc_handle
Aug 12 08:03:14 *** python[17028]:     self._do_on_disconnect(rc, properties)
Aug 12 08:03:14 *** python[17028]:   File "/usr/local/lib/python3.7/dist-packages/paho/mqtt/client.py", line 3360, in _do_on_disconnect
Aug 12 08:03:14 *** python[17028]:     self.on_disconnect(self, self._userdata, rc)
Aug 12 08:03:14 *** python[17028]:   File "/usr/local/lib/python3.7/dist-packages/azure/iot/device/common/mqtt_transport.py", line 210, in on_disconnect
Aug 12 08:03:14 *** python[17028]:     logger.debug("".join(traceback.format_stack()))
Aug 12 08:03:14 *** python[17028]: INFO:azure.iot.device.common.mqtt_transport:Forcing paho disconnect to prevent it from automatically reconnecting
Aug 12 08:03:14 *** python[17028]: DEBUG:azure.iot.device.common.mqtt_transport:in paho thread.  nulling _thread
Aug 12 08:03:14 *** python[17028]: DEBUG:azure.iot.device.common.mqtt_transport:Done forcing paho disconnect
Aug 12 08:03:14 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_thread:Starting _on_mqtt_disconnected in pipeline thread
Aug 12 08:03:14 *** python[17028]: INFO:azure.iot.device.common.pipeline.pipeline_stages_mqtt:MQTTTransportStage: _on_mqtt_disconnect called: ConnectionDroppedError('Paho returned rc==1')
Aug 12 08:03:14 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:ReconnectStage(DisconnectedEvent): State is LOGICALLY_CONNECTED Connected is True.
Aug 12 08:03:14 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:ReconnectStage(DisconnectedEvent): Attempting to reconnect
Aug 12 08:03:14 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:PipelineRootStage: DisconnectedEvent received. Calling on_disconnected_handler
Aug 12 08:03:14 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_thread:Starting _on_disconnected in callback thread
Aug 12 08:03:14 *** python[17028]: INFO:azure.iot.device.common.pipeline.pipeline_stages_mqtt:MQTTTransportStage: disconnection was unexpected
Aug 12 08:03:14 *** python[17028]: INFO:azure.iot.device.iothub.abstract_clients:Connection State - Disconnected
Aug 12 08:03:14 *** python[17028]: INFO:azure.iot.device.iothub.abstract_clients:Cleared all pending method requests due to disconnect
Aug 12 08:03:14 *** python[17028]: INFO:azure.iot.device.common.handle_exceptions:Unexpected disconnection
Aug 12 08:03:14 *** python[17028]: INFO:azure.iot.device.common.handle_exceptions:azure.iot.device.common.transport_exceptions.ConnectionDroppedError: ConnectionDroppedError('Paho returned rc==1')
Aug 12 08:03:14 *** python[17028]: The above exception was the direct cause of the following exception:
Aug 12 08:03:14 *** python[17028]: Traceback (most recent call last):
Aug 12 08:03:14 *** python[17028]:   File "/usr/local/lib/python3.7/dist-packages/azure/iot/device/common/handle_exceptions.py", line 43, in swallow_unraised_exception
Aug 12 08:03:14 *** python[17028]:     raise e
Aug 12 08:03:14 *** python[17028]: azure.iot.device.common.transport_exceptions.ConnectionDroppedError: ConnectionDroppedError(None) caused by ConnectionDroppedError('Paho returned rc==1')
Aug 12 08:03:14 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_thread:Starting on_reconnect_timer_expired in pipeline thread
Aug 12 08:03:14 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:ReconnectStage: Reconnect timer expired. State is WAITING_TO_RECONNECT Connected is False.
Aug 12 08:03:14 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:ReconnectStage: sending new connect op down
Aug 12 08:03:14 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:ConnectionLockStage(ConnectOperation): blocking
Aug 12 08:03:14 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_mqtt:MQTTTransportStage(ConnectOperation): connecting
Aug 12 08:03:14 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_mqtt:MQTTTransportStage(ConnectOperation): Starting watchdog
Aug 12 08:03:14 *** python[17028]: DEBUG:azure.iot.device.common.mqtt_transport:connecting to mqtt broker
Aug 12 08:03:14 *** python[17028]: INFO:azure.iot.device.common.mqtt_transport:Connect using port 8883 (TCP)
Aug 12 08:03:20 *** python[17028]: DEBUG:paho:Sending CONNECT (u1, p1, wr0, wq0, wf0, c0, k60) client_id=b'***'
Aug 12 08:03:20 *** python[17028]: DEBUG:azure.iot.device.common.mqtt_transport:_mqtt_client.connect returned rc=0
Aug 12 08:03:20 *** python[17028]: DEBUG:paho:Received CONNACK (1, 0)
Aug 12 08:03:20 *** python[17028]: INFO:azure.iot.device.common.mqtt_transport:connected with result code: 0
Aug 12 08:03:20 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_thread:Starting _on_mqtt_connected in pipeline thread
Aug 12 08:03:20 *** python[17028]: INFO:azure.iot.device.common.pipeline.pipeline_stages_mqtt:_on_mqtt_connected called
Aug 12 08:03:20 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:PipelineRootStage: ConnectedEvent received. Calling on_connected_handler
Aug 12 08:03:20 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_thread:Starting _on_connected in callback thread
Aug 12 08:03:20 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_mqtt:completing connect op
Aug 12 08:03:20 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_mqtt:MQTTTransportStage(ConnectOperation): cancelling watchdog
Aug 12 08:03:20 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_ops_base:ConnectOperation: completing without error
Aug 12 08:03:20 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:ConnectionLockStage(ConnectOperation): op succeeded.  Unblocking queue
Aug 12 08:03:20 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:ConnectionLockStage(ConnectOperation): unblocking and releasing queued ops.
Aug 12 08:03:20 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:ConnectionLockStage(ConnectOperation): processing 0 items in queue for error=None
Aug 12 08:03:20 *** python[17028]: INFO:azure.iot.device.iothub.abstract_clients:Connection State - Connected
Aug 12 08:03:20 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:ReconnectStage(ConnectOperation): on_connect_complete error=None state=LOGICALLY_CONNECTED never_connected=False connected=True
Aug 12 08:03:20 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:ReconnectStage: completing waiting ops with error=None
Aug 12 08:03:22 *** python[17028]: INFO:azure.iot.device.common.mqtt_transport:disconnected with result code: 1
Aug 12 08:03:22 *** python[17028]: DEBUG:azure.iot.device.common.mqtt_transport:  File "/usr/lib/python3.7/threading.py", line 885, in _bootstrap
Aug 12 08:03:22 *** python[17028]:     self._bootstrap_inner()
Aug 12 08:03:22 *** python[17028]:   File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
Aug 12 08:03:22 *** python[17028]:     self.run()
Aug 12 08:03:22 *** python[17028]:   File "/usr/lib/python3.7/threading.py", line 865, in run
Aug 12 08:03:22 *** python[17028]:     self._target(*self._args, **self._kwargs)
Aug 12 08:03:22 *** python[17028]:   File "/usr/local/lib/python3.7/dist-packages/paho/mqtt/client.py", line 3452, in _thread_main
Aug 12 08:03:22 *** python[17028]:     self.loop_forever(retry_first_connection=True)
Aug 12 08:03:22 *** python[17028]:   File "/usr/local/lib/python3.7/dist-packages/paho/mqtt/client.py", line 1779, in loop_forever
Aug 12 08:03:22 *** python[17028]:     rc = self.loop(timeout, max_packets)
Aug 12 08:03:22 *** python[17028]:   File "/usr/local/lib/python3.7/dist-packages/paho/mqtt/client.py", line 1181, in loop
Aug 12 08:03:22 *** python[17028]:     rc = self.loop_read(max_packets)
Aug 12 08:03:22 *** python[17028]:   File "/usr/local/lib/python3.7/dist-packages/paho/mqtt/client.py", line 1574, in loop_read
Aug 12 08:03:22 *** python[17028]:     return self._loop_rc_handle(rc)
Aug 12 08:03:22 *** python[17028]:   File "/usr/local/lib/python3.7/dist-packages/paho/mqtt/client.py", line 2227, in _loop_rc_handle
Aug 12 08:03:22 *** python[17028]:     self._do_on_disconnect(rc, properties)
Aug 12 08:03:22 *** python[17028]:   File "/usr/local/lib/python3.7/dist-packages/paho/mqtt/client.py", line 3360, in _do_on_disconnect
Aug 12 08:03:22 *** python[17028]:     self.on_disconnect(self, self._userdata, rc)
Aug 12 08:03:22 *** python[17028]:   File "/usr/local/lib/python3.7/dist-packages/azure/iot/device/common/mqtt_transport.py", line 210, in on_disconnect
Aug 12 08:03:22 *** python[17028]:     logger.debug("".join(traceback.format_stack()))
Aug 12 08:03:22 *** python[17028]: INFO:azure.iot.device.common.mqtt_transport:Forcing paho disconnect to prevent it from automatically reconnecting
Aug 12 08:03:22 *** python[17028]: DEBUG:azure.iot.device.common.mqtt_transport:in paho thread.  nulling _thread
Aug 12 08:03:22 *** python[17028]: DEBUG:azure.iot.device.common.mqtt_transport:Done forcing paho disconnect
Aug 12 08:03:22 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_thread:Starting _on_mqtt_disconnected in pipeline thread
Aug 12 08:03:22 *** python[17028]: INFO:azure.iot.device.common.pipeline.pipeline_stages_mqtt:MQTTTransportStage: _on_mqtt_disconnect called: ConnectionDroppedError('Paho returned rc==1')
Aug 12 08:03:22 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:ReconnectStage(DisconnectedEvent): State is LOGICALLY_CONNECTED Connected is True.
Aug 12 08:03:22 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:ReconnectStage(DisconnectedEvent): Attempting to reconnect
Aug 12 08:03:22 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:PipelineRootStage: DisconnectedEvent received. Calling on_disconnected_handler
Aug 12 08:03:22 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_thread:Starting _on_disconnected in callback thread
Aug 12 08:03:22 *** python[17028]: INFO:azure.iot.device.common.pipeline.pipeline_stages_mqtt:MQTTTransportStage: disconnection was unexpected
Aug 12 08:03:22 *** python[17028]: INFO:azure.iot.device.common.handle_exceptions:Unexpected disconnection
Aug 12 08:03:22 *** python[17028]: INFO:azure.iot.device.common.handle_exceptions:azure.iot.device.common.transport_exceptions.ConnectionDroppedError: ConnectionDroppedError('Paho returned rc==1')
Aug 12 08:03:22 *** python[17028]: The above exception was the direct cause of the following exception:
Aug 12 08:03:22 *** python[17028]: Traceback (most recent call last):
Aug 12 08:03:22 *** python[17028]:   File "/usr/local/lib/python3.7/dist-packages/azure/iot/device/common/handle_exceptions.py", line 43, in swallow_unraised_exception
Aug 12 08:03:22 *** python[17028]:     raise e
Aug 12 08:03:22 *** python[17028]: azure.iot.device.common.transport_exceptions.ConnectionDroppedError: ConnectionDroppedError(None) caused by ConnectionDroppedError('Paho returned rc==1')
Aug 12 08:03:22 *** python[17028]: INFO:azure.iot.device.iothub.abstract_clients:Connection State - Disconnected
Aug 12 08:03:22 *** python[17028]: INFO:azure.iot.device.iothub.abstract_clients:Cleared all pending method requests due to disconnect
Aug 12 08:03:22 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_thread:Starting on_reconnect_timer_expired in pipeline thread
Aug 12 08:03:22 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:ReconnectStage: Reconnect timer expired. State is WAITING_TO_RECONNECT Connected is False.
Aug 12 08:03:22 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:ReconnectStage: sending new connect op down
Aug 12 08:03:22 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:ConnectionLockStage(ConnectOperation): blocking
Aug 12 08:03:22 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_mqtt:MQTTTransportStage(ConnectOperation): connecting
Aug 12 08:03:22 *** python[17028]: DEBUG:azure.iot.device.common.pipeline.pipeline_stages_mqtt:MQTTTransportStage(ConnectOperation): Starting watchdog
Aug 12 08:03:22 *** python[17028]: DEBUG:azure.iot.device.common.mqtt_transport:connecting to mqtt broker
Aug 12 08:03:22 *** python[17028]: INFO:azure.iot.device.common.mqtt_transport:Connect using port 8883 (TCP)

Any thoughts, or things I can do to help debug?

Issue Analytics

State:
Created 2 years ago
Comments:10 (7 by maintainers)

Top GitHub Comments

2reactions

cartertinneycommented, Aug 25, 2021

@kodonnell

Apologies for the delay, but getting this next release has been slower than expected for various reasons. I’m still actively working on this next patch that will address your issue (among others) as my top priority. I would hope to get it out around the beginning of the month, but I can’t say for sure yet.

0reactions

cartertinneycommented, Oct 12, 2021

@kodonnell

Please do. We are focusing on this area of reliability right now, and ramping up our testing scenarios, so any issues you have will be a top priority.

I’ll close this issue now, and if you have any further ones, please open a new issue

Top Results From Across the Web

Troubleshooting Azure IoT Hub error codes | Microsoft Learn

Understand how to fix errors reported by Azure IoT Hub. ... IoT Hub doesn't allow more than one client to connect using the...

KEPServerEX and Microsoft Azure IoT Hub - Kepware

This Connectivity Guide discusses how to configure the KEPServerEX® IoT Gateway Plug-in to communicate with Microsoft® Azure IoT Hub using MQTT over TLS....

Connect to Azure IoT Hub - IoT DevKit - Microsoft Open Source

In this project, you will create an Azure IoT Hub, connect IoT DevKit to it, and collect the temperature and humidity data from...

Integrate into Azure IoT Hub - Seeed Wiki

You can connect millions of devices and their backend solutions reliably and securely. Almost any device can be connected to an IoT hub....

azure iot hub - Disconnect (or Close) device connection if no ...

Unfortunately there is no such feature available in Azure IoTHub. You can manage it programmatically. You can keep the device in sleep mode ......