Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Connecting to EdgeHub fails after several successful messages

See original GitHub issue
  • OS and version used: Windows 11, 21H2 (OS Build 22000.493)
  • Python version: Python 3.10.2
  • pip version: pip 22.0.4
  • list of installed packages:
Package                    Version
-------------------------- ---------
altgraph                   0.17.2
applicationinsights        0.11.10
atomicwrites               1.4.0
attrs                      21.4.0
azure-iot-device           2.11.0
azure-iot-edge-runtime-ctl 1.0.0rc17
bcrypt                     3.2.0
cached-property            1.5.2
certifi                    2021.10.8
cffi                       1.15.0
chardet                    3.0.4
charset-normalizer         2.0.11
click                      8.0.3
colorama                   0.4.4
cryptography               36.0.1
decorator                  5.1.1
deprecation                2.1.0
docker                     2.6.0
docker-compose             1.24.0
docker-pycreds             0.4.0
dockerpty                  0.4.1
docopt                     0.6.2
future                     0.18.2
idna                       2.7
iniconfig                  1.1.1
iotedgehubdev              0.11.1
janus                      1.0.0
jsonpath-rw                1.4.0
jsonschema                 2.6.0
packaging                  21.3
paho-mqtt                  1.6.1
paramiko                   2.9.2
pefile                     2021.9.3
pip                        22.0.4
pluggy                     1.0.0
ply                        3.11
psutil                     5.9.0
py                         1.11.0
pycparser                  2.21
pyinstaller                4.9
pyinstaller-hooks-contrib  2022.2
PyNaCl                     1.5.0
pyOpenSSL                  21.0.0
pyparsing                  3.0.7
pypiwin32                  223
PySocks                    1.7.1
pytest                     7.0.1
python-dotenv              0.19.2
python-slugify             6.1.1
pywin32                    303
pywin32-ctypes             0.2.0
PyYAML                     4.2b4
requests                   2.27.1
requests-unixsocket        0.3.0
setuptools                 60.5.0
six                        1.16.0
text-unidecode             1.3
texttable                  0.9.1
tomli                      2.0.1
typing_extensions          4.0.1
urllib3                    1.26.8
websocket-client           0.59.0
wheel                      0.37.1

Context

  • Azure IoT Hub
  • Azure IoT Edge device
    • Ubuntu 20.04.4 LTS
    • IoT Edge version 1.2.9
    • Symmetric Key Authentication
    • Installed certificates to enable communication with IoT device
  • Azure IoT device
    • Python on Windows
    • Symmetric Key Autenthication
    • Uses MQTT over WS

This is new issue, branching off from this one: https://github.com/Azure/azure-iot-sdk-python/issues/990#issuecomment-1112724888

Current behaviour

I’m using the sample here to simulate an IoT device on the windows machine, with only the connection string and logging setup changed: https://github.com/Azure/azure-iot-sdk-python/blob/main/azure-iot-device/samples/simple_send_message.py I can run the script and send a message successfully. I can do this several times (the number varies between 3 and 7 or 8 successful attempts). After that, the script cannot connect anymore.

What I’m also observing, but I’m not sure if it is the same issue: I have another script that sends trivial messages through the same Edge Hub. It runs fine for several hours, but stops working without warning. I have modified it that it restarts on exceptions, and usually the messages can get through to the hub after several hours of downtime without.

I did setup a completely new edgeHub to see if it is an issue with the old Edge device, but the error also happens with the new one.

Expected behaviour

I can run the script successfully an unlimited number of time

Evidence

Python logs

Logfile of the device client with the last successful connect (ends at Client shutdown complete) and the subscequent failing connect attempt.


2022-05-03 08:59:20,223 [DEBUG] Using proactor: IocpProactor
2022-05-03 08:59:20,224 [DEBUG] Starting run_op in pipeline thread
2022-05-03 08:59:20,225 [DEBUG] Creating pipeline executor
2022-05-03 08:59:20,226 [DEBUG] Gateway Hostname Present. Setting Hostname to: [hidden hostname]
2022-05-03 08:59:20,226 [DEBUG] HTTPTransportStage(InitializePipelineOperation): got connection args
2022-05-03 08:59:20,226 [DEBUG] creating a SSL context
2022-05-03 08:59:20,240 [DEBUG] InitializePipelineOperation: completing without error
2022-05-03 08:59:20,240 [DEBUG] Starting <azure.iot.device.common.evented_callback.EventedCallback object at 0x000001A45D919600> in callback thread
2022-05-03 08:59:20,240 [DEBUG] Creating callback executor
2022-05-03 08:59:20,241 [DEBUG] Callback completed with result None
2022-05-03 08:59:20,241 [DEBUG] Starting run_op in pipeline thread
2022-05-03 08:59:20,241 [DEBUG] SasTokenStage: Scheduling automatic SAS Token renewal at epoch time: 1651564640
2022-05-03 08:59:20,242 [DEBUG] Gateway Hostname Present. Setting Hostname to: [hidden hostname]
2022-05-03 08:59:20,242 [DEBUG] MQTTTransportStage(InitializePipelineOperation): got connection args
2022-05-03 08:59:20,242 [DEBUG] creating mqtt client
2022-05-03 08:59:20,242 [INFO] Creating client for connecting using MQTT over TCP
2022-05-03 08:59:20,242 [DEBUG] creating a SSL context
2022-05-03 08:59:20,243 [DEBUG] configuring SSL context with default certs
2022-05-03 08:59:20,251 [DEBUG] Created MQTT protocol client, assigned callbacks
2022-05-03 08:59:20,252 [DEBUG] InitializePipelineOperation: completing without error
2022-05-03 08:59:20,252 [DEBUG] Starting <azure.iot.device.common.evented_callback.EventedCallback object at 0x000001A45D91A4D0> in callback thread
2022-05-03 08:59:20,252 [DEBUG] Callback completed with result None
2022-05-03 08:59:20,252 [DEBUG] Creating new event loop - CLIENT_INTERNAL_LOOP
2022-05-03 08:59:20,252 [DEBUG] Using proactor: IocpProactor
2022-05-03 08:59:20,255 [INFO] Connecting to Hub...
2022-05-03 08:59:20,256 [DEBUG] Starting ConnectOperation on the pipeline
2022-05-03 08:59:20,256 [DEBUG] Starting run_op in pipeline thread
2022-05-03 08:59:20,256 [DEBUG] ReconnectStage(ConnectOperation): State changes DISCONNECTED -> CONNECTING. Sending op down
2022-05-03 08:59:20,256 [DEBUG] ConnectionLockStage(ConnectOperation): blocking
2022-05-03 08:59:20,256 [DEBUG] MQTTTransportStage(ConnectOperation): connecting
2022-05-03 08:59:20,256 [DEBUG] MQTTTransportStage(ConnectOperation): Starting watchdog
2022-05-03 08:59:20,257 [DEBUG] connecting to mqtt broker
2022-05-03 08:59:20,257 [INFO] Connect using port 8883 (TCP)
2022-05-03 08:59:20,347 [DEBUG] Sending CONNECT (u1, p1, wr0, wq0, wf0, c0, k60) client_id=b'python-test-device'
2022-05-03 08:59:20,347 [DEBUG] _mqtt_client.connect returned rc=0
2022-05-03 08:59:20,354 [DEBUG] Received CONNACK (0, 0)
2022-05-03 08:59:20,354 [INFO] connected with result code: 0
2022-05-03 08:59:20,354 [DEBUG] Starting _on_mqtt_connected in pipeline thread
2022-05-03 08:59:20,354 [INFO] _on_mqtt_connected called
2022-05-03 08:59:20,354 [DEBUG] ReconnectStage(ConnectedEvent): State changes CONNECTING -> CONNECTED. Connection established
2022-05-03 08:59:20,354 [DEBUG] PipelineRootStage: ConnectedEvent received. Calling on_connected_handler
2022-05-03 08:59:20,354 [DEBUG] Starting _on_connected in callback thread
2022-05-03 08:59:20,354 [DEBUG] MQTTTransportStage: completing connect op
2022-05-03 08:59:20,355 [INFO] Connection State - Connected
2022-05-03 08:59:20,355 [DEBUG] MQTTTransportStage(ConnectOperation): cancelling watchdog
2022-05-03 08:59:20,355 [DEBUG] ConnectOperation: completing without error
2022-05-03 08:59:20,355 [DEBUG] ConnectionLockStage(ConnectOperation): op succeeded.  Unblocking queue
2022-05-03 08:59:20,355 [DEBUG] ConnectionLockStage(ConnectOperation): unblocking and releasing queued ops.
2022-05-03 08:59:20,355 [DEBUG] ConnectionLockStage(ConnectOperation): processing 0 items in queue for error=None
2022-05-03 08:59:20,355 [DEBUG] Starting on_complete in callback thread
2022-05-03 08:59:20,356 [DEBUG] Callback completed with result None
2022-05-03 08:59:20,356 [INFO] Successfully connected to Hub
2022-05-03 08:59:20,356 [INFO] Sending message to Hub...
2022-05-03 08:59:20,357 [DEBUG] Starting SendD2CMessageOperation on the pipeline
2022-05-03 08:59:20,357 [DEBUG] Starting run_op in pipeline thread
2022-05-03 08:59:20,357 [DEBUG] SendD2CMessageOperation: creating worker op of type MQTTPublishOperation
2022-05-03 08:59:20,357 [DEBUG] MQTTTransportStage(MQTTPublishOperation): publishing on devices/python-test-device/messages/events/
2022-05-03 08:59:20,357 [INFO] publishing on devices/python-test-device/messages/events/
2022-05-03 08:59:20,358 [DEBUG] Sending PUBLISH (d0, q1, r0, m1), 'b'devices/python-test-device/messages/events/'', ... (36 bytes)
2022-05-03 08:59:20,358 [DEBUG] _mqtt_client.publish returned rc=0
2022-05-03 08:59:20,359 [DEBUG] Waiting for response on MID: 1
2022-05-03 08:59:20,364 [DEBUG] Received PUBACK (Mid: 1)
2022-05-03 08:59:20,364 [INFO] payload published for 1
2022-05-03 08:59:20,364 [DEBUG] Response received for recognized MID: 1 - triggering callback
2022-05-03 08:59:20,364 [DEBUG] Starting on_complete in pipeline thread
2022-05-03 08:59:20,364 [DEBUG] MQTTTransportStage(MQTTPublishOperation): PUBACK received. completing op.
2022-05-03 08:59:20,364 [DEBUG] MQTTPublishOperation: completing without error
2022-05-03 08:59:20,364 [DEBUG] SendD2CMessageOperation: Worker op (MQTTPublishOperation) has been completed
2022-05-03 08:59:20,364 [DEBUG] SendD2CMessageOperation: completing without error
2022-05-03 08:59:20,364 [DEBUG] Starting on_complete in callback thread
2022-05-03 08:59:20,365 [DEBUG] Callback completed with result None
2022-05-03 08:59:20,365 [INFO] Successfully sent message to Hub
2022-05-03 08:59:20,366 [INFO] Initiating client shutdown
2022-05-03 08:59:20,366 [INFO] Disconnecting from Hub...
2022-05-03 08:59:20,366 [DEBUG] Executing initial disconnect
2022-05-03 08:59:20,366 [DEBUG] Starting DisconnectOperation on the pipeline
2022-05-03 08:59:20,366 [DEBUG] Starting run_op in pipeline thread
2022-05-03 08:59:20,366 [DEBUG] ReconnectStage(DisconnectOperation): State changes CONNECTED -> DISCONNECTING. Sending op down.
2022-05-03 08:59:20,367 [DEBUG] ConnectionLockStage(DisconnectOperation): blocking
2022-05-03 08:59:20,367 [DEBUG] MQTTTransportStage(DisconnectOperation): disconnecting
2022-05-03 08:59:20,367 [INFO] disconnecting MQTT client
2022-05-03 08:59:20,367 [DEBUG] Sending DISCONNECT
2022-05-03 08:59:20,367 [INFO] disconnected with result code: 0
2022-05-03 08:59:20,367 [DEBUG] Starting _on_mqtt_disconnected in pipeline thread
2022-05-03 08:59:20,368 [DEBUG] _mqtt_client.disconnect returned rc=0
2022-05-03 08:59:20,368 [DEBUG] Cancelling all pending operations
2022-05-03 08:59:20,368 [INFO] MQTTTransportStage: _on_mqtt_disconnect called
2022-05-03 08:59:20,368 [DEBUG] ReconnectStage(DisconnectedEvent): State changes DISCONNECTING -> DISCONNECTED. Not attempting to reconnect (User-initiated disconnect)
2022-05-03 08:59:20,369 [DEBUG] PipelineRootStage: DisconnectedEvent received. Calling on_disconnected_handler
2022-05-03 08:59:20,369 [DEBUG] Starting _on_disconnected in callback thread
2022-05-03 08:59:20,369 [DEBUG] MQTTTransportStage: Expected disconnect - completing pending disconnect op
2022-05-03 08:59:20,369 [INFO] Connection State - Disconnected
2022-05-03 08:59:20,369 [DEBUG] DisconnectOperation: completing without error
2022-05-03 08:59:20,369 [INFO] Cleared all pending method requests due to disconnect
2022-05-03 08:59:20,369 [DEBUG] ConnectionLockStage(DisconnectOperation): op succeeded.  Unblocking queue
2022-05-03 08:59:20,370 [DEBUG] ConnectionLockStage(DisconnectOperation): unblocking and releasing queued ops.
2022-05-03 08:59:20,370 [DEBUG] ConnectionLockStage(DisconnectOperation): processing 0 items in queue for error=None
2022-05-03 08:59:20,370 [DEBUG] Starting on_complete in callback thread
2022-05-03 08:59:20,370 [DEBUG] Callback completed with result None
2022-05-03 08:59:20,370 [DEBUG] Successfully executed initial disconnect
2022-05-03 08:59:20,370 [DEBUG] Stopping handlers...
2022-05-03 08:59:20,370 [DEBUG] Successfully stopped handlers
2022-05-03 08:59:20,370 [DEBUG] Executing secondary disconnect...
2022-05-03 08:59:20,371 [DEBUG] Starting DisconnectOperation on the pipeline
2022-05-03 08:59:20,371 [DEBUG] Starting run_op in pipeline thread
2022-05-03 08:59:20,371 [DEBUG] ReconnectStage(DisconnectOperation): State is already DISCONNECTED. Sending op down
2022-05-03 08:59:20,371 [INFO] ConnectionLockStage(DisconnectOperation): Transport is already disconnected.  Completing.
2022-05-03 08:59:20,371 [DEBUG] DisconnectOperation: completing without error
2022-05-03 08:59:20,371 [DEBUG] Starting on_complete in callback thread
2022-05-03 08:59:20,371 [DEBUG] Callback completed with result None
2022-05-03 08:59:20,372 [DEBUG] Successfully executed secondary disconnect
2022-05-03 08:59:20,372 [INFO] Successfully disconnected from Hub
2022-05-03 08:59:20,372 [DEBUG] Beginning pipeline shutdown operation
2022-05-03 08:59:20,372 [DEBUG] Commencing shutdown of pipeline
2022-05-03 08:59:20,372 [DEBUG] Starting run_op in pipeline thread
2022-05-03 08:59:20,372 [DEBUG] Cancelling SAS Token update alarm
2022-05-03 08:59:20,373 [INFO] Forcing paho disconnect to prevent it from automatically reconnecting
2022-05-03 08:59:20,373 [DEBUG] Done forcing paho disconnect
2022-05-03 08:59:20,373 [DEBUG] Cancelling all pending operations
2022-05-03 08:59:20,373 [DEBUG] ShutdownPipelineOperation: completing without error
2022-05-03 08:59:20,373 [DEBUG] Starting on_complete in callback thread
2022-05-03 08:59:20,373 [DEBUG] Callback completed with result None
2022-05-03 08:59:20,374 [DEBUG] Completed pipeline shutdown operation
2022-05-03 08:59:20,374 [INFO] Client shutdown complete
2022-05-03 08:59:24,422 [DEBUG] Using proactor: IocpProactor
2022-05-03 08:59:24,422 [DEBUG] Starting run_op in pipeline thread
2022-05-03 08:59:24,422 [DEBUG] Creating pipeline executor
2022-05-03 08:59:24,422 [DEBUG] Gateway Hostname Present. Setting Hostname to: [hidden hostname]
2022-05-03 08:59:24,422 [DEBUG] HTTPTransportStage(InitializePipelineOperation): got connection args
2022-05-03 08:59:24,422 [DEBUG] creating a SSL context
2022-05-03 08:59:24,444 [DEBUG] InitializePipelineOperation: completing without error
2022-05-03 08:59:24,444 [DEBUG] Starting <azure.iot.device.common.evented_callback.EventedCallback object at 0x000001D662119600> in callback thread
2022-05-03 08:59:24,444 [DEBUG] Creating callback executor
2022-05-03 08:59:24,444 [DEBUG] Callback completed with result None
2022-05-03 08:59:24,445 [DEBUG] Starting run_op in pipeline thread
2022-05-03 08:59:24,445 [DEBUG] SasTokenStage: Scheduling automatic SAS Token renewal at epoch time: 1651564644
2022-05-03 08:59:24,446 [DEBUG] Gateway Hostname Present. Setting Hostname to: [hidden hostname]
2022-05-03 08:59:24,446 [DEBUG] MQTTTransportStage(InitializePipelineOperation): got connection args
2022-05-03 08:59:24,446 [DEBUG] creating mqtt client
2022-05-03 08:59:24,446 [INFO] Creating client for connecting using MQTT over TCP
2022-05-03 08:59:24,446 [DEBUG] creating a SSL context
2022-05-03 08:59:24,446 [DEBUG] configuring SSL context with default certs
2022-05-03 08:59:24,454 [DEBUG] Created MQTT protocol client, assigned callbacks
2022-05-03 08:59:24,454 [DEBUG] InitializePipelineOperation: completing without error
2022-05-03 08:59:24,454 [DEBUG] Starting <azure.iot.device.common.evented_callback.EventedCallback object at 0x000001D66211A4D0> in callback thread
2022-05-03 08:59:24,454 [DEBUG] Callback completed with result None
2022-05-03 08:59:24,455 [DEBUG] Creating new event loop - CLIENT_INTERNAL_LOOP
2022-05-03 08:59:24,455 [DEBUG] Using proactor: IocpProactor
2022-05-03 08:59:24,457 [INFO] Connecting to Hub...
2022-05-03 08:59:24,458 [DEBUG] Starting ConnectOperation on the pipeline
2022-05-03 08:59:24,458 [DEBUG] Starting run_op in pipeline thread
2022-05-03 08:59:24,458 [DEBUG] ReconnectStage(ConnectOperation): State changes DISCONNECTED -> CONNECTING. Sending op down
2022-05-03 08:59:24,458 [DEBUG] ConnectionLockStage(ConnectOperation): blocking
2022-05-03 08:59:24,459 [DEBUG] MQTTTransportStage(ConnectOperation): connecting
2022-05-03 08:59:24,459 [DEBUG] MQTTTransportStage(ConnectOperation): Starting watchdog
2022-05-03 08:59:24,459 [DEBUG] connecting to mqtt broker
2022-05-03 08:59:24,459 [INFO] Connect using port 8883 (TCP)
2022-05-03 09:00:24,460 [DEBUG] Starting watchdog_function in pipeline thread
2022-05-03 09:00:24,476 [INFO] Forcing paho disconnect to prevent it from automatically reconnecting
2022-05-03 09:00:24,476 [DEBUG] Done forcing paho disconnect
2022-05-03 09:00:24,476 [INFO] transport.connect raised error
2022-05-03 09:00:24,479 [INFO] Traceback (most recent call last):
  File "C:\Users\pebneter\AppData\Local\Programs\Python\Python310\lib\site-packages\azure\iot\device\common\mqtt_transport.py", line 390, in connect
    rc = self._mqtt_client.connect(
  File "C:\Users\pebneter\AppData\Local\Programs\Python\Python310\lib\site-packages\paho\mqtt\client.py", line 914, in connect
    return self.reconnect()
  File "C:\Users\pebneter\AppData\Local\Programs\Python\Python310\lib\site-packages\paho\mqtt\client.py", line 1073, in reconnect
    sock.do_handshake()
  File "C:\Users\pebneter\AppData\Local\Programs\Python\Python310\lib\ssl.py", line 1341, in do_handshake
    self._sslobj.do_handshake()
TimeoutError: _ssl.c:980: The handshake operation timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\pebneter\AppData\Local\Programs\Python\Python310\lib\site-packages\azure\iot\device\common\pipeline\pipeline_stages_mqtt.py", line 189, in _run_op
    self.transport.connect(password=password)
  File "C:\Users\pebneter\AppData\Local\Programs\Python\Python310\lib\site-packages\azure\iot\device\common\mqtt_transport.py", line 413, in connect
    raise exceptions.ConnectionFailedError() from e
azure.iot.device.common.transport_exceptions.ConnectionFailedError

2022-05-03 09:00:24,479 [DEBUG] MQTTTransportStage(ConnectOperation): cancelling watchdog
2022-05-03 09:00:24,479 [DEBUG] ConnectOperation: completing with error 
2022-05-03 09:00:24,479 [DEBUG] ConnectionLockStage(ConnectOperation): op failed.  Unblocking queue with error: 
2022-05-03 09:00:24,479 [DEBUG] ConnectionLockStage(ConnectOperation): unblocking and releasing queued ops.
2022-05-03 09:00:24,479 [DEBUG] ConnectionLockStage(ConnectOperation): processing 0 items in queue for error=
2022-05-03 09:00:24,479 [DEBUG] ReconnectStage(ConnectOperation): failed, state change CONNECTING -> DISCONNECTED
2022-05-03 09:00:24,479 [DEBUG] Starting on_complete in callback thread
2022-05-03 09:00:24,479 [DEBUG] Connection watchdog expired, but pending op is not the same op
2022-05-03 09:00:24,480 [INFO] Callback completed with error 
2022-05-03 09:00:24,480 [INFO] ['azure.iot.device.common.transport_exceptions.ConnectionFailedError\n']

EdgeHub logs

It seems like the second attempt is not even showing up in these logs

<7> 2022-05-03 06:59:20.349 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.IDeviceScopeIdentitiesCache] - Getting service identity for python-test-device
<6> 2022-05-03 06:59:20.349 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Hub.Core.DeviceScopeAuthenticator] - Client python-test-device in device scope authenticated locally.
<7> 2022-05-03 06:59:20.349 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Authenticator] - Client python-test-device authentication successful
<6> 2022-05-03 06:59:20.350 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Hub.Mqtt.DeviceIdentityProvider] - Successfully generated identity for clientId python-test-device and username iothubrag4.azure-devices.net/python-test-device/?api-version=2019-10-01&DeviceClientType=azure-iot-device-iothub-py%2F2.11.0%283.10.2%3BWindows%2010.0.22000%3BAMD64%29
<6> 2022-05-03 06:59:20.352 +00:00 [INF] [EdgeHub] - ClientAuthenticated, python-test-device, 0b2c609d
<6> 2022-05-03 06:59:20.357 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Hub.Core.ConnectionManager] - New device connection for device python-test-device
<6> 2022-05-03 06:59:20.357 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Hub.Core.SubscriptionProcessor] - Client python-test-device connected to edgeHub, processing existing subscriptions.
<7> 2022-05-03 06:59:20.357 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.ConnectionManager] - Obtained cloud connection for device python-test-device
<7> 2022-05-03 06:59:20.357 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.EdgeHubConnection] - Updating device python-test-device connection status to Connected
<7> 2022-05-03 06:59:20.358 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Twin.StoringTwinManager] - Updating reported properties for connectivity-poc-edge-device-01/$edgeHub
<6> 2022-05-03 06:59:20.358 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Hub.Core.Device.DeviceMessageHandler] - Bind device proxy for device python-test-device
<6> 2022-05-03 06:59:20.358 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Hub.Mqtt.MessagingServiceClient] - Binding message channel for device Id python-test-device
<7> 2022-05-03 06:59:20.359 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Mqtt.MessageAddressConverter] - Message address devices/python-test-device/messages/events/ matches more than one topic. Picking first matching topic.
<7> 2022-05-03 06:59:20.359 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Mqtt.MessagingServiceClient] - Processing message for device Id python-test-device
<7> 2022-05-03 06:59:20.359 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Routing.RoutingEdgeHub] - Received message from python-test-device
<7> 2022-05-03 06:59:20.360 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Storage.MessageStore] - Getting next batch for endpoint iothub starting from 27220 with batch size 100.
<7> 2022-05-03 06:59:20.360 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Storage.MessageStore] - Obtained next batch for endpoint iothub with batch size 1. Next start offset = 27221.
<7> 2022-05-03 06:59:20.361 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Storage.MessageStore] - Getting next batch for endpoint iothub starting from 27221 with batch size 99.
<7> 2022-05-03 06:59:20.361 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Storage.MessageStore] - Obtained next batch for endpoint iothub with batch size 0. Next start offset = 27221.
<7> 2022-05-03 06:59:20.361 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Routing.CloudEndpoint] - Sending 1 message(s) upstream.
<7> 2022-05-03 06:59:20.361 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Routing.CloudEndpoint] - Sending 1 message(s) upstream, divided into 1 groups. Processing maximum 10 groups in parallel.
<7> 2022-05-03 06:59:20.361 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.ConnectionManager] - Obtained cloud connection for device python-test-device
<7> 2022-05-03 06:59:20.362 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x5c8e98a1, [::ffff:172.18.0.3]:39778 => [::ffff:40.113.176.174]:8883] WRITE: PublishPacket[Type=PUBLISH, QualityOfService=AtLeastOnce, Duplicate=False, Retain=False], 36B
         +-------------------------------------------------+
         |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
+--------+-------------------------------------------------+----------------+
|100000000| 54 68 69 73 20 69 73 20 61 20 6D 65 73 73 61 67 |This is a messag|
|100000010| 65 20 74 68 61 74 20 69 73 20 62 65 69 6E 67 20 |e that is being |
|100000020| 73 65 6E 74                                     |sent            |
+--------+-------------------------------------------------+----------------+
<7> 2022-05-03 06:59:20.362 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x5c8e98a1, [::ffff:172.18.0.3]:39778 => [::ffff:40.113.176.174]:8883] FLUSH
<7> 2022-05-03 06:59:20.362 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x5c8e98a1, [::ffff:172.18.0.3]:39778 => [::ffff:40.113.176.174]:8883] FLUSH
<7> 2022-05-03 06:59:20.362 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Routing.CloudEndpoint] - Finished processing messages to upstream
<6> 2022-05-03 06:59:20.367 +00:00 [INF] [EdgeHub] - Closing connection for device: python-test-device, ,
<6> 2022-05-03 06:59:20.368 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Hub.Mqtt.MessagingServiceClient] - Disposing MessagingServiceClient for device Id python-test-device because of exception -
<6> 2022-05-03 06:59:20.368 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Hub.Mqtt.DeviceProxy] - Setting device proxy inactive for device Id python-test-device
<6> 2022-05-03 06:59:20.368 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Hub.Core.ConnectionManager] - Removing device connection for device python-test-device with removeCloudConnection flag 'True'.
<7> 2022-05-03 06:59:20.368 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x5c8e98a1, [::ffff:172.18.0.3]:39778 => [::ffff:40.113.176.174]:8883] WRITE: DisconnectPacket[Type=DISCONNECT, QualityOfService=AtMostOnce, Duplicate=False, Retain=False]
<7> 2022-05-03 06:59:20.368 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient] - Received connection status changed callback with connection status Disabled and reason Client_Close for python-test-device
<7> 2022-05-03 06:59:20.368 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x5c8e98a1, [::ffff:172.18.0.3]:39778 => [::ffff:40.113.176.174]:8883] FLUSH
<7> 2022-05-03 06:59:20.485 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x5c8e98a1, [::ffff:172.18.0.3]:39778 => [::ffff:40.113.176.174]:8883] RECEIVED_COMPLETE
<7> 2022-05-03 06:59:20.485 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x5c8e98a1, [::ffff:172.18.0.3]:39778 => [::ffff:40.113.176.174]:8883] READ
<7> 2022-05-03 06:59:20.485 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x5c8e98a1, [::ffff:172.18.0.3]:39778 => [::ffff:40.113.176.174]:8883] READ
<7> 2022-05-03 06:59:20.486 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x5c8e98a1, [::ffff:172.18.0.3]:39778 :> [::ffff:40.113.176.174]:8883] INACTIVE
<7> 2022-05-03 06:59:20.486 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x5c8e98a1, [::ffff:172.18.0.3]:39778 :> [::ffff:40.113.176.174]:8883] CLOSE
<7> 2022-05-03 06:59:20.486 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x5c8e98a1, [::ffff:172.18.0.3]:39778 :> [::ffff:40.113.176.174]:8883] UNREGISTERED
<7> 2022-05-03 06:59:20.489 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x5c8e98a1, [::ffff:172.18.0.3]:39778 :> [::ffff:40.113.176.174]:8883] HANDLER_REMOVED
<7> 2022-05-03 06:59:21.367 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient] - Operation SendEventAsync failed for python-test-device
System.InvalidOperationException: Invalid transport state: Closed
   at Microsoft.Azure.Devices.Client.Transport.Mqtt.MqttTransportHandler.EnsureValidState(Boolean throwIfNotOpen)
   at Microsoft.Azure.Devices.Client.Transport.Mqtt.MqttTransportHandler.SendEventAsync(Message message, CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.<>c__DisplayClass27_0.<<ExecuteWithErrorHandlingAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.<>c__DisplayClass15_0.<<SendEventAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.SendEventAsync(Message message, CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.InternalClient.SendEventAsync(Message message)
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass30_0.<<InvokeFunc>b__0>d.MoveNext() in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 194
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 145
<7> 2022-05-03 06:59:21.377 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudProxy] - Error sending message for python-test-device in cloud proxy 26e74931-94f4-47cf-b0f6-f8a0d98bdb0d
System.InvalidOperationException: Invalid transport state: Closed
   at Microsoft.Azure.Devices.Client.Transport.Mqtt.MqttTransportHandler.EnsureValidState(Boolean throwIfNotOpen)
   at Microsoft.Azure.Devices.Client.Transport.Mqtt.MqttTransportHandler.SendEventAsync(Message message, CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.<>c__DisplayClass27_0.<<ExecuteWithErrorHandlingAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.<>c__DisplayClass15_0.<<SendEventAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.SendEventAsync(Message message, CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.InternalClient.SendEventAsync(Message message)
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass30_0.<<InvokeFunc>b__0>d.MoveNext() in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 194
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 145
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 181
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudProxy.SendMessageAsync(IMessage inputMessage) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudProxy.cs:line 157
<6> 2022-05-03 06:59:21.379 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Hub.Core.Cloud.RetryingCloudProxy] - Retrying cloud proxy operation SendMessageAsync for python-test-device.
System.InvalidOperationException: Invalid transport state: Closed
   at Microsoft.Azure.Devices.Client.Transport.Mqtt.MqttTransportHandler.EnsureValidState(Boolean throwIfNotOpen)
   at Microsoft.Azure.Devices.Client.Transport.Mqtt.MqttTransportHandler.SendEventAsync(Message message, CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.<>c__DisplayClass27_0.<<ExecuteWithErrorHandlingAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.<>c__DisplayClass15_0.<<SendEventAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.SendEventAsync(Message message, CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.InternalClient.SendEventAsync(Message message)
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass30_0.<<InvokeFunc>b__0>d.MoveNext() in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 194
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 145
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 181
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudProxy.SendMessageAsync(IMessage inputMessage) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudProxy.cs:line 157
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudProxy.SendMessageAsync(IMessage inputMessage) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudProxy.cs:line 165
   at Microsoft.Azure.Devices.Edge.Hub.Core.Cloud.RetryingCloudProxy.<>c__DisplayClass24_0.<<ExecuteOperation>b__0>d.MoveNext() in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.Core/cloud/RetryingCloudProxy.cs:line 64
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Edge.Hub.Core.Cloud.RetryingCloudProxy.ExecuteOperation[T](Func`2 func, String operation) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.Core/cloud/RetryingCloudProxy.cs:line 76
<7> 2022-05-03 06:59:21.380 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Cloud.RetryingCloudProxy] - Getting new cloud proxy for client python-test-device since current cloud proxy is not active
<7> 2022-05-03 06:59:21.380 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.IDeviceScopeIdentitiesCache] - Getting service identity for python-test-device
<7> 2022-05-03 06:59:21.380 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection] - Creating cloud connection for client python-test-device using EdgeHub credentials
<6> 2022-05-03 06:59:21.381 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection] - Attempting to connect to IoT Hub for client python-test-device via MQTT...
<7> 2022-05-03 06:59:21.413 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x1f6d6dcd] HANDLER_ADDED
<7> 2022-05-03 06:59:21.413 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x1f6d6dcd] REGISTERED
<7> 2022-05-03 06:59:21.413 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x1f6d6dcd] CONNECT: 40.113.176.174:8883
<7> 2022-05-03 06:59:21.437 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x1f6d6dcd, [::ffff:172.18.0.3]:39780 => [::ffff:40.113.176.174]:8883] ACTIVE
<7> 2022-05-03 06:59:21.438 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x1f6d6dcd, [::ffff:172.18.0.3]:39780 => [::ffff:40.113.176.174]:8883] WRITE: ConnectPacket[Type=CONNECT, QualityOfService=AtMostOnce, Duplicate=False, Retain=False]
<7> 2022-05-03 06:59:21.438 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x1f6d6dcd, [::ffff:172.18.0.3]:39780 => [::ffff:40.113.176.174]:8883] FLUSH
<7> 2022-05-03 06:59:21.439 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x1f6d6dcd, [::ffff:172.18.0.3]:39780 => [::ffff:40.113.176.174]:8883] READ
<7> 2022-05-03 06:59:21.467 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x1f6d6dcd, [::ffff:172.18.0.3]:39780 => [::ffff:40.113.176.174]:8883] RECEIVED_COMPLETE
<7> 2022-05-03 06:59:21.467 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x1f6d6dcd, [::ffff:172.18.0.3]:39780 => [::ffff:40.113.176.174]:8883] READ
<7> 2022-05-03 06:59:21.467 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x1f6d6dcd, [::ffff:172.18.0.3]:39780 => [::ffff:40.113.176.174]:8883] READ
<7> 2022-05-03 06:59:21.468 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x1f6d6dcd, [::ffff:172.18.0.3]:39780 => [::ffff:40.113.176.174]:8883] RECEIVED_COMPLETE
<7> 2022-05-03 06:59:21.468 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x1f6d6dcd, [::ffff:172.18.0.3]:39780 => [::ffff:40.113.176.174]:8883] READ
<7> 2022-05-03 06:59:21.468 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x1f6d6dcd, [::ffff:172.18.0.3]:39780 => [::ffff:40.113.176.174]:8883] READ
<7> 2022-05-03 06:59:21.494 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x1f6d6dcd, [::ffff:172.18.0.3]:39780 => [::ffff:40.113.176.174]:8883] USER_EVENT: TlsHandshakeCompletionEvent(SUCCESS)
<7> 2022-05-03 06:59:21.496 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x1f6d6dcd, [::ffff:172.18.0.3]:39780 => [::ffff:40.113.176.174]:8883] READ
<7> 2022-05-03 06:59:21.497 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x1f6d6dcd, [::ffff:172.18.0.3]:39780 => [::ffff:40.113.176.174]:8883] RECEIVED_COMPLETE
<7> 2022-05-03 06:59:21.497 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x1f6d6dcd, [::ffff:172.18.0.3]:39780 => [::ffff:40.113.176.174]:8883] READ
<7> 2022-05-03 06:59:21.497 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x1f6d6dcd, [::ffff:172.18.0.3]:39780 => [::ffff:40.113.176.174]:8883] READ
<7> 2022-05-03 06:59:21.548 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x1f6d6dcd, [::ffff:172.18.0.3]:39780 => [::ffff:40.113.176.174]:8883] RECEIVED: ConnAckPacket[Type=CONNACK, QualityOfService=AtMostOnce, Duplicate=False, Retain=False]
<7> 2022-05-03 06:59:21.549 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x1f6d6dcd, [::ffff:172.18.0.3]:39780 => [::ffff:40.113.176.174]:8883] WRITE: SubscribePacket[Type=SUBSCRIBE, QualityOfService=AtLeastOnce, Duplicate=False, Retain=False]
<7> 2022-05-03 06:59:21.549 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x1f6d6dcd, [::ffff:172.18.0.3]:39780 => [::ffff:40.113.176.174]:8883] FLUSH
<7> 2022-05-03 06:59:21.549 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x1f6d6dcd, [::ffff:172.18.0.3]:39780 => [::ffff:40.113.176.174]:8883] RECEIVED_COMPLETE
<7> 2022-05-03 06:59:21.549 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x1f6d6dcd, [::ffff:172.18.0.3]:39780 => [::ffff:40.113.176.174]:8883] READ
<7> 2022-05-03 06:59:21.549 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x1f6d6dcd, [::ffff:172.18.0.3]:39780 => [::ffff:40.113.176.174]:8883] READ
<7> 2022-05-03 06:59:21.550 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x1f6d6dcd, [::ffff:172.18.0.3]:39780 => [::ffff:40.113.176.174]:8883] WRITE: SubscribePacket[Type=SUBSCRIBE, QualityOfService=AtLeastOnce, Duplicate=False, Retain=False]
<7> 2022-05-03 06:59:21.550 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x1f6d6dcd, [::ffff:172.18.0.3]:39780 => [::ffff:40.113.176.174]:8883] FLUSH
<7> 2022-05-03 06:59:21.587 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x1f6d6dcd, [::ffff:172.18.0.3]:39780 => [::ffff:40.113.176.174]:8883] RECEIVED: SubAckPacket[Type=SUBACK, QualityOfService=AtMostOnce, Duplicate=False, Retain=False]
<7> 2022-05-03 06:59:21.587 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x1f6d6dcd, [::ffff:172.18.0.3]:39780 => [::ffff:40.113.176.174]:8883] RECEIVED_COMPLETE
<7> 2022-05-03 06:59:21.587 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x1f6d6dcd, [::ffff:172.18.0.3]:39780 => [::ffff:40.113.176.174]:8883] READ
<7> 2022-05-03 06:59:21.587 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x1f6d6dcd, [::ffff:172.18.0.3]:39780 => [::ffff:40.113.176.174]:8883] READ
<7> 2022-05-03 06:59:21.600 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x1f6d6dcd, [::ffff:172.18.0.3]:39780 => [::ffff:40.113.176.174]:8883] RECEIVED: SubAckPacket[Type=SUBACK, QualityOfService=AtMostOnce, Duplicate=False, Retain=False]
<7> 2022-05-03 06:59:21.600 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x1f6d6dcd, [::ffff:172.18.0.3]:39780 => [::ffff:40.113.176.174]:8883] RECEIVED_COMPLETE
<7> 2022-05-03 06:59:21.600 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x1f6d6dcd, [::ffff:172.18.0.3]:39780 => [::ffff:40.113.176.174]:8883] READ
<7> 2022-05-03 06:59:21.600 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x1f6d6dcd, [::ffff:172.18.0.3]:39780 => [::ffff:40.113.176.174]:8883] READ
<7> 2022-05-03 06:59:21.601 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient] - Received connection status changed callback with connection status Connected and reason Connection_Ok for python-test-device
<7> 2022-05-03 06:59:21.601 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceConnectivityManager] - IotHub call succeeded
<6> 2022-05-03 06:59:21.605 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient] - Cloud connection for python-test-device is True
<6> 2022-05-03 06:59:21.605 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Hub.Core.ConnectionManager] - Connection status for python-test-device changed to ConnectionEstablished
<7> 2022-05-03 06:59:21.605 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.ConnectionManager] - Invoking cloud connection established event for python-test-device
<6> 2022-05-03 06:59:21.605 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Hub.Core.SubscriptionProcessor] - Client python-test-device connected to cloud, processing existing subscriptions.
<7> 2022-05-03 06:59:21.605 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient] - Operation OpenAsync succeeded for python-test-device
<6> 2022-05-03 06:59:21.605 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection] - Created cloud proxy for client python-test-device via MQTT, with client operation timeout 20 seconds.
<6> 2022-05-03 06:59:21.606 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudProxy] - Initialized cloud proxy 6b39b089-3f50-4100-9593-738fbefdcf83 for python-test-device
<6> 2022-05-03 06:59:21.606 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection] - Created cloud connection for client python-test-device
<7> 2022-05-03 06:59:21.606 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.ConnectionManager] - Obtained cloud connection for device python-test-device
<7> 2022-05-03 06:59:21.608 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.ConnectionManager] - Obtained cloud connection for device python-test-device
<7> 2022-05-03 06:59:21.608 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Cloud.RetryingCloudProxy] - Get new cloud proxy for client python-test-device
<7> 2022-05-03 06:59:21.613 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x1f6d6dcd, [::ffff:172.18.0.3]:39780 => [::ffff:40.113.176.174]:8883] WRITE: PublishPacket[Type=PUBLISH, QualityOfService=AtLeastOnce, Duplicate=False, Retain=False], 36B
         +-------------------------------------------------+
         |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
+--------+-------------------------------------------------+----------------+
|100000000| 54 68 69 73 20 69 73 20 61 20 6D 65 73 73 61 67 |This is a messag|
|100000010| 65 20 74 68 61 74 20 69 73 20 62 65 69 6E 67 20 |e that is being |
|100000020| 73 65 6E 74                                     |sent            |
+--------+-------------------------------------------------+----------------+
<7> 2022-05-03 06:59:21.613 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x1f6d6dcd, [::ffff:172.18.0.3]:39780 => [::ffff:40.113.176.174]:8883] FLUSH
<7> 2022-05-03 06:59:21.613 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x1f6d6dcd, [::ffff:172.18.0.3]:39780 => [::ffff:40.113.176.174]:8883] FLUSH
<7> 2022-05-03 06:59:21.908 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x1f6d6dcd, [::ffff:172.18.0.3]:39780 => [::ffff:40.113.176.174]:8883] RECEIVED: PubAckPacket[Type=PUBACK, QualityOfService=AtMostOnce, Duplicate=False, Retain=False]
<7> 2022-05-03 06:59:21.908 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x1f6d6dcd, [::ffff:172.18.0.3]:39780 => [::ffff:40.113.176.174]:8883] RECEIVED_COMPLETE
<7> 2022-05-03 06:59:21.908 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x1f6d6dcd, [::ffff:172.18.0.3]:39780 => [::ffff:40.113.176.174]:8883] READ
<7> 2022-05-03 06:59:21.908 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x1f6d6dcd, [::ffff:172.18.0.3]:39780 => [::ffff:40.113.176.174]:8883] READ
<7> 2022-05-03 06:59:21.909 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceConnectivityManager] - IotHub call succeeded
<7> 2022-05-03 06:59:21.909 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient] - Operation SendEventAsync succeeded for python-test-device
<7> 2022-05-03 06:59:21.913 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudProxy] - Sending message for python-test-device
<7> 2022-05-03 06:59:21.914 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Storage.MessageStore] - Getting next batch for endpoint iothub starting from 27221 with batch size 100.
<7> 2022-05-03 06:59:21.914 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Storage.MessageStore] - Obtained next batch for endpoint iothub with batch size 0. Next start offset = 27221.
<7> 2022-05-03 06:59:26.124 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x4597ef45, [::ffff:172.18.0.3]:39744 => [::ffff:40.113.176.174]:8883] WRITE: PingReqPacket[Type=PINGREQ, QualityOfService=AtMostOnce, Duplicate=False, Retain=False]
<7> 2022-05-03 06:59:26.124 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x4597ef45, [::ffff:172.18.0.3]:39744 => [::ffff:40.113.176.174]:8883] FLUSH
<7> 2022-05-03 06:59:26.149 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x4597ef45, [::ffff:172.18.0.3]:39744 => [::ffff:40.113.176.174]:8883] RECEIVED: PingRespPacket[Type=PINGRESP, QualityOfService=AtMostOnce, Duplicate=False, Retain=False]
<7> 2022-05-03 06:59:26.149 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x4597ef45, [::ffff:172.18.0.3]:39744 => [::ffff:40.113.176.174]:8883] RECEIVED_COMPLETE
<7> 2022-05-03 06:59:26.149 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x4597ef45, [::ffff:172.18.0.3]:39744 => [::ffff:40.113.176.174]:8883] READ
<7> 2022-05-03 06:59:26.149 +00:00 [DBG] [DotNetty.Handlers.Logging.LoggingHandler] - [id: 0x4597ef45, [::ffff:172.18.0.3]:39744 => [::ffff:40.113.176.174]:8883] READ
<7> 2022-05-03 06:59:31.912 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Storage.MessageStore] - Getting next batch for endpoint iothub starting from 27221 with batch size 100.
<7> 2022-05-03 06:59:31.912 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Storage.MessageStore] - Obtained next batch for endpoint iothub with batch size 0. Next start offset = 27221.
<7> 2022-05-03 06:59:41.915 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Storage.MessageStore] - Getting next batch for endpoint iothub starting from 27221 with batch size 100.
<7> 2022-05-03 06:59:41.915 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Storage.MessageStore] - Obtained next batch for endpoint iothub with batch size 0. Next start offset = 27221.
<7> 2022-05-03 06:59:51.918 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Storage.MessageStore] - Getting next batch for endpoint iothub starting from 27221 with batch size 100.
<7> 2022-05-03 06:59:51.918 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Storage.MessageStore] - Obtained next batch for endpoint iothub with batch size 0. Next start offset = 27221.
<7> 2022-05-03 07:00:01.920 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Storage.MessageStore] - Getting next batch for endpoint iothub starting from 27221 with batch size 100.
<7> 2022-05-03 07:00:01.920 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Storage.MessageStore] - Obtained next batch for endpoint iothub with batch size 0. Next start offset = 27221.
<7> 2022-05-03 07:00:11.918 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Storage.MessageStore] - Getting next batch for endpoint iothub starting from 27221 with batch size 100.
<7> 2022-05-03 07:00:11.918 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Storage.MessageStore] - Obtained next batch for endpoint iothub with batch size 0. Next start offset = 27221.
<7> 2022-05-03 07:00:21.922 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Storage.MessageStore] - Getting next batch for endpoint iothub starting from 27221 with batch size 100.
<7> 2022-05-03 07:00:21.922 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Storage.MessageStore] - Obtained next batch for endpoint iothub with batch size 0. Next start offset = 27221.
<7> 2022-05-03 07:00:31.921 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Storage.MessageStore] - Getting next batch for endpoint iothub starting from 27221 with batch size 100.
<7> 2022-05-03 07:00:31.922 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Storage.MessageStore] - Obtained next batch for endpoint iothub with batch size 0. Next start offset = 27221.

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:20 (5 by maintainers)

github_iconTop GitHub Comments

1reaction
lucarvcommented, Sep 28, 2022

You need to reopen this issue. Upgrading the platform to 1.4 will not solve the problem. I suspect that with the changes introduced in 1.2 (with the new identity service) something has corrupted the re-authentication flow between a python module and edgeHub.

@gauravIoTEdge and @jlian for comments please.

1reaction
pebnetercommented, May 25, 2022

@cartertinney Yes has been working flawlessly since downgrading to 1.1

Read more comments on GitHub >

github_iconTop Results From Across the Web

Troubleshoot Azure IoT Edge common errors - Microsoft Learn
The agent attempts to connect over AMQP (port 5671) first. ... IoT Edge module fails to send a message to edgeHub with 404...
Read more >
Losing Messages after Timeout · Issue #1613 · Azure/iotedge
As far as we can tell, when the timeout error occurs the message that edgeHub was trying to send is not persisted or...
Read more >
azure - IoT Edge module fails to send a message to edgeHub ...
The custom module was successfully sending messages to the edge hub till IoT Edge runtime version 1.2.10. This issue was noticed in a...
Read more >
EdgeRouter X con't connect for more than 30-60 seconds
The connection to the router was lost/unreachable. After resetting, it said I was on 1.8.5. Attempting to run a wizard quickly gives some...
Read more >
KEPServerEX and Microsoft Azure IoT Edge - Kepware
part of TLS negotiation at the beginning of the KEPServerEX MQTT connection sequence to the. Edge Hub. TLS negotiation will fail unless the...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Thread leakage and unhandled exception due to constant disconnects

Next page

Unhandled exception in Paho `_thread_main` leaves the device client in a falsely 'connected' state