Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Possible No Recovery after ConnectionDroppedError (aio IoTHubDeviceClient)
See original GitHub issueContext
- OS and version used: Ubuntu 18.04
- Python version: Python 3.7.5
- pip version: 21.0.1
- list of installed packages: Package Version
aiohttp 3.7.4.post0 asn1crypto 0.24.0 async-timeout 3.0.1 attrs 20.3.0 Automat 0.6.0 azure-core 1.13.0 azure-identity 1.5.0 azure-iot-device 2.5.1 azure-storage-blob 12.8.0 blinker 1.4 certifi 2020.12.5 cffi 1.14.5 chardet 4.0.0 click 6.7 cloud-init 20.4 colorama 0.3.7 command-not-found 0.3 configobj 5.0.6 constantly 15.1.0 cpppo 4.0.4 cryptography 3.4.7 deprecation 2.1.0 distro-info 0.18ubuntu0.18.04.1 fastavro 1.2.4 greenery 2.1 httplib2 0.9.2 hyperlink 17.3.1 idna 2.10 incremental 16.10.1 ipaddress 1.0.23 isodate 0.6.0 janus 0.4.0 Jinja2 2.10 jsonpatch 1.16 jsonpointer 1.10 jsonschema 2.6.0 keyring 10.6.0 keyrings.alt 3.0 language-selector 0.1 MarkupSafe 1.0 msal 1.11.0 msal-extensions 0.3.0 msrest 0.6.21 mtc-datareader 2.0.5.dev3 mtc-iot 2.1.0 multidict 5.1.0 netifaces 0.10.4 oauthlib 3.1.0 packaging 20.9 paho-mqtt 1.5.1 PAM 0.4.2 pexpect 4.2.1 pip 21.0.1 portalocker 1.7.1 pyasn1 0.4.2 pyasn1-modules 0.2.1 pycparser 2.20 pycrypto 2.6.1 pygobject 3.26.1 PyJWT 2.0.1 pylogix 0.7.5 pymodbus 2.3.0 pyOpenSSL 17.5.0 pyparsing 2.4.7 pyserial 3.5 PySocks 1.7.1 python-apt 1.6.5+ubuntu0.5 python-debian 0.1.32 pytz 2021.1 pyxdg 0.25 PyYAML 3.12 requests 2.25.1 requests-oauthlib 1.3.0 requests-unixsocket 0.2.0 SecretStorage 2.3.1 service-identity 16.0.0 setuptools 39.0.1 six 1.15.0 sos 4.1 ssh-import-id 5.7 systemd-python 234 Twisted 17.9.0 typing-extensions 3.7.4.3 ufw 0.36 unattended-upgrades 0.1 urllib3 1.25.11 wheel 0.30.0 yarl 1.6.3 zope.interface 4.3.2
Description of the issue
Let me preface the following information with the following: the issue I’m documenting here is hard to reproduce as it seemingly occurs at random times. I believe we have seen this issue pop up periodically in our device fleet, although we didn’t have the logs verbose enough to get much information from the failures. It is certainly possible the issue is instead with our code, or with edgeHub. I won’t be able to give you a code sample that reliably reproduces. But, I could provide one that interacts with all the same APIs in the same way as our code. It would take me some time to produce a pared down version of what we’re doing, and at a point it may just be simpler to privately share our source code. I figure if the logs I provide make certain aspects seem more likely to be the root cause than others then I will do a deep-dive in that direction to the extent possible, rather than spending the time to develop a pared down version upfront.
One more preface: We’ve designed our device code as a systemd service that fails fast when unexpected issues occur. In other words, we don’t catch exceptions unless we’re absolutely certain we can recover from them. The systemd service is setup to restart if it fails, so generally we recover from unexpected, transient issues in a matter of seconds or minutes.
At (seemingly) random moments and intervals, our leaf device stops sending messages to edgeHub (we’re in a gateway scenario). When it stops sending messages the entire system comes to a halt, but does not fail either (if it failed we would have recovered). Our device code uses a producer/consumer design to model things. In our case, the “producers” are different types of industrial networking clients that read sensor data. The “consumers” are coroutines that have a reference to our single IoTHubDeviceClient instance. We have a 1:1 ratio of producers to consumers. What appears to happen when data stops coming in to our IoT Edge device is all the “consumers” get stuck trying to send a message, so our system comes to a halt.
In theory our “producers” could be causing problems, but they would still get some small number of events out per minute if things were going wrong. The reason I say this is we run our producers as background task, and have a separate task that monitors them and restarts them in the case of an error. Plus, even when producers are failing they generate error messages that we also send as telemetry to edgeHub.
What is especially strange about the outage we had is a workaround that we’ve put in place failed to resolve the issue. The workaround is a simple cron job that periodically (every 2 hours) restarts our iot device systemd service. We put this workaround in place to stem the issue that we have seen previously, which has the exact same behavior as the issue being documented here (mentioned in the first paragraph), but in this case it did not resolve the issue at all. In other words, restarting the system service running our device code did not resolve the outage. It wasn’t until a team member rebooted the device at the OS level that we recovered from the outage. The team member also rebooted our IoT Edge gateway device, so it’s unfortunately hard to say if rebooting one or the other (or the combo) resolved the issue.
Code sample exhibiting the issue
(please let me know if a code sample would be helpful based on the above description and the logs below).
Console log of the issue
For reference, the outage occurred around 10:29 UTC (can be seen in ASA logs at bottom) and didn’t recover until 13:44 UTC after rebooting at 13:39 UTC. You can see at 12:01 UTC in the Leaf Device logs our periodic cron job to restart the iot device system service.
Leaf Device Logs:
Apr 15 10:01:01 datareader0899 systemd[1]: Started dataReader service.
Apr 15 10:59:01 datareader0899 run-datareader[13241]: 2021-04-15 10:59:01,002 - azure.iot.device.common.pipeline.pipeline_stages_base - WARNING - Unexpected error in <azure.iot.device.common.pipeline.pipeline_stages_mqtt.MQTTTransportStage object at 0x7f8270a510d0>._run_op() call
Apr 15 10:59:01 datareader0899 run-datareader[13241]: WARNING:azure.iot.device.common.pipeline.pipeline_stages_base:Unexpected error in <azure.iot.device.common.pipeline.pipeline_stages_mqtt.MQTTTransportStage object at 0x7f8270a510d0>._run_op() call
Apr 15 10:59:01 datareader0899 run-datareader[13241]: 2021-04-15 10:59:01,003 - azure.iot.device.common.pipeline.pipeline_stages_base - WARNING - Traceback (most recent call last):
Apr 15 10:59:01 datareader0899 run-datareader[13241]: File "/home/iotadmin/.local/lib/python3.7/site-packages/azure/iot/device/common/pipeline/pipeline_stages_base.py", line 103, in run_op
Apr 15 10:59:01 datareader0899 run-datareader[13241]: self._run_op(op)
Apr 15 10:59:01 datareader0899 run-datareader[13241]: File "/home/iotadmin/.local/lib/python3.7/site-packages/azure/iot/device/common/pipeline/pipeline_thread.py", line 203, in wrapper
Apr 15 10:59:01 datareader0899 run-datareader[13241]: return func(*args, **kwargs)
Apr 15 10:59:01 datareader0899 run-datareader[13241]: File "/home/iotadmin/.local/lib/python3.7/site-packages/azure/iot/device/common/pipeline/pipeline_stages_mqtt.py", line 229, in _run_op
Apr 15 10:59:01 datareader0899 run-datareader[13241]: self.transport.publish(topic=op.topic, payload=op.payload, callback=on_published)
Apr 15 10:59:01 datareader0899 run-datareader[13241]: File "/home/iotadmin/.local/lib/python3.7/site-packages/azure/iot/device/common/mqtt_transport.py", line 555, in publish
Apr 15 10:59:01 datareader0899 run-datareader[13241]: raise _create_error_from_rc_code(rc)
Apr 15 10:59:01 datareader0899 run-datareader[13241]: azure.iot.device.common.transport_exceptions.ConnectionDroppedError: ConnectionDroppedError('The client is not currently connected.')
Apr 15 10:59:01 datareader0899 run-datareader[13241]: WARNING:azure.iot.device.common.pipeline.pipeline_stages_base:Traceback (most recent call last):
Apr 15 10:59:01 datareader0899 run-datareader[13241]: File "/home/iotadmin/.local/lib/python3.7/site-packages/azure/iot/device/common/pipeline/pipeline_stages_base.py", line 103, in run_op
Apr 15 10:59:01 datareader0899 run-datareader[13241]: self._run_op(op)
Apr 15 10:59:01 datareader0899 run-datareader[13241]: File "/home/iotadmin/.local/lib/python3.7/site-packages/azure/iot/device/common/pipeline/pipeline_thread.py", line 203, in wrapper
Apr 15 10:59:01 datareader0899 run-datareader[13241]: return func(*args, **kwargs)
Apr 15 10:59:01 datareader0899 run-datareader[13241]: File "/home/iotadmin/.local/lib/python3.7/site-packages/azure/iot/device/common/pipeline/pipeline_stages_mqtt.py", line 229, in _run_op
Apr 15 10:59:01 datareader0899 run-datareader[13241]: self.transport.publish(topic=op.topic, payload=op.payload, callback=on_published)
Apr 15 10:59:01 datareader0899 run-datareader[13241]: File "/home/iotadmin/.local/lib/python3.7/site-packages/azure/iot/device/common/mqtt_transport.py", line 555, in publish
Apr 15 10:59:01 datareader0899 run-datareader[13241]: raise _create_error_from_rc_code(rc)
Apr 15 10:59:01 datareader0899 run-datareader[13241]: azure.iot.device.common.transport_exceptions.ConnectionDroppedError: ConnectionDroppedError('The client is not currently connected.')
Apr 15 12:01:01 datareader0899 systemd[1]: Stopping dataReader service...
Apr 15 12:01:01 datareader0899 run-datareader[13241]: Received exit signal Signals.SIGTERM...
Apr 15 12:01:01 datareader0899 run-datareader[13241]: Clearing out messages in message buffer
Apr 15 12:01:01 datareader0899 run-datareader[13241]: Cancelling outstanding tasks
Apr 15 12:01:01 datareader0899 run-datareader[13241]: Producer cancelled. Closing and exiting.
Apr 15 12:01:01 datareader0899 run-datareader[13241]: Cancellation received in run()
Apr 15 12:01:01 datareader0899 run-datareader[13241]: Starting shutdown procedure
Apr 15 12:01:01 datareader0899 run-datareader[13241]: Producer cancelled. Closing and exiting.
Apr 15 12:01:01 datareader0899 run-datareader[13241]: Exiting.
Apr 15 12:01:01 datareader0899 run-datareader[13241]: ERROR:asyncio:Task was destroyed but it is pending!
Apr 15 12:01:01 datareader0899 run-datareader[13241]: task: <Task pending coro=<AsyncHandlerManager._inbox_handler_runner() running at /home/iotadmin/.local/lib/python3.7/site-packages/azure/iot/device/iothub/aio/async_handler_manager.py:65> wait_for=<Future pending cb=[_chain_future.<locals>._call_check_cancel() at /usr/lib/python3.7/asyncio/futures.py:348, <TaskWakeupMethWrapper object at 0x7f827019e6d0>()]> cb=[_chain_future.<locals>._call_set_state() at /usr/lib/python3.7/asyncio/futures.py:355]>
Apr 15 12:01:01 datareader0899 run-datareader[13241]: ERROR:asyncio:Task was destroyed but it is pending!
Apr 15 12:01:01 datareader0899 run-datareader[13241]: task: <Task pending coro=<_AsyncQueueProxy.get() running at /home/iotadmin/.local/lib/python3.7/site-packages/janus/__init__.py:451> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f8270a65b50>()]> cb=[_chain_future.<locals>._call_set_state() at /usr/lib/python3.7/asyncio/futures.py:355]>
Apr 15 12:01:01 datareader0899 systemd[1]: Stopped dataReader service.
Apr 15 12:01:01 datareader0899 systemd[1]: Started dataReader service.
Apr 15 13:39:05 datareader0899 systemd[1]: Stopping dataReader service...
Apr 15 13:39:05 datareader0899 run-datareader[20577]: Received exit signal Signals.SIGTERM...
Apr 15 13:39:05 datareader0899 run-datareader[20577]: Clearing out messages in message buffer
Apr 15 13:39:05 datareader0899 run-datareader[20577]: Cancelling outstanding tasks
Apr 15 13:39:05 datareader0899 run-datareader[20577]: Producer cancelled. Closing and exiting.
Apr 15 13:39:05 datareader0899 run-datareader[20577]: Cancellation received in run()
Apr 15 13:39:05 datareader0899 run-datareader[20577]: Starting shutdown procedure
Apr 15 13:39:05 datareader0899 run-datareader[20577]: Producer cancelled. Closing and exiting.
Apr 15 13:39:05 datareader0899 run-datareader[20577]: Exiting.
Apr 15 13:39:05 datareader0899 run-datareader[20577]: 2021-04-15 13:39:05,084 - azure.iot.device.common.mqtt_transport - ERROR - Unexpected error calling callback for MID: 21669
Apr 15 13:39:05 datareader0899 run-datareader[20577]: ERROR:azure.iot.device.common.mqtt_transport:Unexpected error calling callback for MID: 21669
Apr 15 13:39:05 datareader0899 run-datareader[20577]: 2021-04-15 13:39:05,095 - azure.iot.device.common.mqtt_transport - ERROR - Traceback (most recent call last):
Apr 15 13:39:05 datareader0899 run-datareader[20577]: File "/home/iotadmin/.local/lib/python3.7/site-packages/azure/iot/device/common/mqtt_transport.py", line 650, in complete_operation
Apr 15 13:39:05 datareader0899 run-datareader[20577]: callback()
Apr 15 13:39:05 datareader0899 run-datareader[20577]: File "/home/iotadmin/.local/lib/python3.7/site-packages/azure/iot/device/common/pipeline/pipeline_thread.py", line 134, in wrapper
Apr 15 13:39:05 datareader0899 run-datareader[20577]: future = _get_named_executor(thread_name).submit(thread_proc)
Apr 15 13:39:05 datareader0899 run-datareader[20577]: File "/usr/lib/python3.7/concurrent/futures/thread.py", line 163, in submit
Apr 15 13:39:05 datareader0899 run-datareader[20577]: raise RuntimeError('cannot schedule new futures after shutdown')
Apr 15 13:39:05 datareader0899 run-datareader[20577]: RuntimeError: cannot schedule new futures after shutdown
Apr 15 13:39:05 datareader0899 run-datareader[20577]: ERROR:azure.iot.device.common.mqtt_transport:Traceback (most recent call last):
Apr 15 13:39:05 datareader0899 run-datareader[20577]: File "/home/iotadmin/.local/lib/python3.7/site-packages/azure/iot/device/common/mqtt_transport.py", line 650, in complete_operation
Apr 15 13:39:05 datareader0899 run-datareader[20577]: callback()
Apr 15 13:39:05 datareader0899 run-datareader[20577]: File "/home/iotadmin/.local/lib/python3.7/site-packages/azure/iot/device/common/pipeline/pipeline_thread.py", line 134, in wrapper
Apr 15 13:39:05 datareader0899 run-datareader[20577]: future = _get_named_executor(thread_name).submit(thread_proc)
Apr 15 13:39:05 datareader0899 run-datareader[20577]: File "/usr/lib/python3.7/concurrent/futures/thread.py", line 163, in submit
Apr 15 13:39:05 datareader0899 run-datareader[20577]: raise RuntimeError('cannot schedule new futures after shutdown')
Apr 15 13:39:05 datareader0899 run-datareader[20577]: RuntimeError: cannot schedule new futures after shutdown
Apr 15 13:39:05 datareader0899 run-datareader[20577]: 2021-04-15 13:39:05,095 - azure.iot.device.common.mqtt_transport - ERROR - Unexpected error calling callback for MID: 21670
Apr 15 13:39:05 datareader0899 run-datareader[20577]: ERROR:azure.iot.device.common.mqtt_transport:Unexpected error calling callback for MID: 21670
Apr 15 13:39:05 datareader0899 run-datareader[20577]: 2021-04-15 13:39:05,096 - azure.iot.device.common.mqtt_transport - ERROR - Traceback (most recent call last):
Apr 15 13:39:05 datareader0899 run-datareader[20577]: File "/home/iotadmin/.local/lib/python3.7/site-packages/azure/iot/device/common/mqtt_transport.py", line 650, in complete_operation
Apr 15 13:39:05 datareader0899 run-datareader[20577]: callback()
Apr 15 13:39:05 datareader0899 run-datareader[20577]: File "/home/iotadmin/.local/lib/python3.7/site-packages/azure/iot/device/common/pipeline/pipeline_thread.py", line 134, in wrapper
Apr 15 13:39:05 datareader0899 run-datareader[20577]: future = _get_named_executor(thread_name).submit(thread_proc)
Apr 15 13:39:05 datareader0899 run-datareader[20577]: File "/usr/lib/python3.7/concurrent/futures/thread.py", line 163, in submit
Apr 15 13:39:05 datareader0899 run-datareader[20577]: raise RuntimeError('cannot schedule new futures after shutdown')
Apr 15 13:39:05 datareader0899 run-datareader[20577]: RuntimeError: cannot schedule new futures after shutdown
Apr 15 13:39:05 datareader0899 run-datareader[20577]: ERROR:azure.iot.device.common.mqtt_transport:Traceback (most recent call last):
Apr 15 13:39:05 datareader0899 run-datareader[20577]: File "/home/iotadmin/.local/lib/python3.7/site-packages/azure/iot/device/common/mqtt_transport.py", line 650, in complete_operation
Apr 15 13:39:05 datareader0899 run-datareader[20577]: callback()
Apr 15 13:39:05 datareader0899 run-datareader[20577]: File "/home/iotadmin/.local/lib/python3.7/site-packages/azure/iot/device/common/pipeline/pipeline_thread.py", line 134, in wrapper
Apr 15 13:39:05 datareader0899 run-datareader[20577]: future = _get_named_executor(thread_name).submit(thread_proc)
Apr 15 13:39:05 datareader0899 run-datareader[20577]: File "/usr/lib/python3.7/concurrent/futures/thread.py", line 163, in submit
Apr 15 13:39:05 datareader0899 run-datareader[20577]: raise RuntimeError('cannot schedule new futures after shutdown')
Apr 15 13:39:05 datareader0899 run-datareader[20577]: RuntimeError: cannot schedule new futures after shutdown
Apr 15 13:39:05 datareader0899 run-datareader[20577]: 2021-04-15 13:39:05,099 - azure.iot.device.common.mqtt_transport - ERROR - Unexpected error calling callback for MID: 21671
Apr 15 13:39:05 datareader0899 run-datareader[20577]: ERROR:azure.iot.device.common.mqtt_transport:Unexpected error calling callback for MID: 21671
Apr 15 13:39:05 datareader0899 run-datareader[20577]: 2021-04-15 13:39:05,099 - azure.iot.device.common.mqtt_transport - ERROR - Traceback (most recent call last):
Apr 15 13:39:05 datareader0899 run-datareader[20577]: File "/home/iotadmin/.local/lib/python3.7/site-packages/azure/iot/device/common/mqtt_transport.py", line 650, in complete_operation
Apr 15 13:39:05 datareader0899 run-datareader[20577]: callback()
Apr 15 13:39:05 datareader0899 run-datareader[20577]: File "/home/iotadmin/.local/lib/python3.7/site-packages/azure/iot/device/common/pipeline/pipeline_thread.py", line 134, in wrapper
Apr 15 13:39:05 datareader0899 run-datareader[20577]: future = _get_named_executor(thread_name).submit(thread_proc)
Apr 15 13:39:05 datareader0899 run-datareader[20577]: File "/usr/lib/python3.7/concurrent/futures/thread.py", line 163, in submit
Apr 15 13:39:05 datareader0899 run-datareader[20577]: raise RuntimeError('cannot schedule new futures after shutdown')
Apr 15 13:39:05 datareader0899 run-datareader[20577]: RuntimeError: cannot schedule new futures after shutdown
Apr 15 13:39:05 datareader0899 run-datareader[20577]: ERROR:azure.iot.device.common.mqtt_transport:Traceback (most recent call last):
Apr 15 13:39:05 datareader0899 run-datareader[20577]: File "/home/iotadmin/.local/lib/python3.7/site-packages/azure/iot/device/common/mqtt_transport.py", line 650, in complete_operation
Apr 15 13:39:05 datareader0899 run-datareader[20577]: callback()
Apr 15 13:39:05 datareader0899 run-datareader[20577]: File "/home/iotadmin/.local/lib/python3.7/site-packages/azure/iot/device/common/pipeline/pipeline_thread.py", line 134, in wrapper
Apr 15 13:39:05 datareader0899 run-datareader[20577]: future = _get_named_executor(thread_name).submit(thread_proc)
Apr 15 13:39:05 datareader0899 run-datareader[20577]: File "/usr/lib/python3.7/concurrent/futures/thread.py", line 163, in submit
Apr 15 13:39:05 datareader0899 run-datareader[20577]: raise RuntimeError('cannot schedule new futures after shutdown')
Apr 15 13:39:05 datareader0899 run-datareader[20577]: RuntimeError: cannot schedule new futures after shutdown
Apr 15 13:39:05 datareader0899 run-datareader[20577]: 2021-04-15 13:39:05,100 - azure.iot.device.common.mqtt_transport - ERROR - Unexpected error calling callback for MID: 21672
Apr 15 13:39:05 datareader0899 run-datareader[20577]: ERROR:azure.iot.device.common.mqtt_transport:Unexpected error calling callback for MID: 21672
Apr 15 13:39:05 datareader0899 run-datareader[20577]: 2021-04-15 13:39:05,103 - azure.iot.device.common.mqtt_transport - ERROR - Traceback (most recent call last):
Apr 15 13:39:05 datareader0899 run-datareader[20577]: File "/home/iotadmin/.local/lib/python3.7/site-packages/azure/iot/device/common/mqtt_transport.py", line 650, in complete_operation
Apr 15 13:39:05 datareader0899 run-datareader[20577]: callback()
Apr 15 13:39:05 datareader0899 run-datareader[20577]: File "/home/iotadmin/.local/lib/python3.7/site-packages/azure/iot/device/common/pipeline/pipeline_thread.py", line 134, in wrapper
Apr 15 13:39:05 datareader0899 run-datareader[20577]: future = _get_named_executor(thread_name).submit(thread_proc)
Apr 15 13:39:05 datareader0899 run-datareader[20577]: File "/usr/lib/python3.7/concurrent/futures/thread.py", line 163, in submit
Apr 15 13:39:05 datareader0899 run-datareader[20577]: raise RuntimeError('cannot schedule new futures after shutdown')
Apr 15 13:39:05 datareader0899 run-datareader[20577]: RuntimeError: cannot schedule new futures after shutdown
Apr 15 13:39:05 datareader0899 run-datareader[20577]: ERROR:azure.iot.device.common.mqtt_transport:Traceback (most recent call last):
Apr 15 13:39:05 datareader0899 run-datareader[20577]: File "/home/iotadmin/.local/lib/python3.7/site-packages/azure/iot/device/common/mqtt_transport.py", line 650, in complete_operation
Apr 15 13:39:05 datareader0899 run-datareader[20577]: callback()
Apr 15 13:39:05 datareader0899 run-datareader[20577]: File "/home/iotadmin/.local/lib/python3.7/site-packages/azure/iot/device/common/pipeline/pipeline_thread.py", line 134, in wrapper
Apr 15 13:39:05 datareader0899 run-datareader[20577]: future = _get_named_executor(thread_name).submit(thread_proc)
Apr 15 13:39:05 datareader0899 run-datareader[20577]: File "/usr/lib/python3.7/concurrent/futures/thread.py", line 163, in submit
Apr 15 13:39:05 datareader0899 run-datareader[20577]: raise RuntimeError('cannot schedule new futures after shutdown')
Apr 15 13:39:05 datareader0899 run-datareader[20577]: RuntimeError: cannot schedule new futures after shutdown
Apr 15 13:39:05 datareader0899 run-datareader[20577]: ERROR:asyncio:Task was destroyed but it is pending!
Apr 15 13:39:05 datareader0899 run-datareader[20577]: task: <Task pending coro=<AsyncHandlerManager._inbox_handler_runner() running at /home/iotadmin/.local/lib/python3.7/site-packages/azure/iot/device/iothub/aio/async_handler_manager.py:65> wait_for=<Future pending cb=[_chain_future.<locals>._call_check_cancel() at /usr/lib/python3.7/asyncio/futures.py:348, <TaskWakeupMethWrapper object at 0x7f149c794d10>()]> cb=[_chain_future.<locals>._call_set_state() at /usr/lib/python3.7/asyncio/futures.py:355]>
Apr 15 13:39:05 datareader0899 run-datareader[20577]: ERROR:asyncio:Task was destroyed but it is pending!
Apr 15 13:39:05 datareader0899 run-datareader[20577]: task: <Task pending coro=<_AsyncQueueProxy.get() running at /home/iotadmin/.local/lib/python3.7/site-packages/janus/__init__.py:451> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f149c7b1490>()]> cb=[_chain_future.<locals>._call_set_state() at /usr/lib/python3.7/asyncio/futures.py:355]>
Apr 15 13:39:05 datareader0899 systemd[1]: Stopped dataReader service.
-- Reboot --
Apr 15 13:39:38 datareader0899 systemd[1]: Started dataReader service.
Apr 15 13:43:36 datareader0899 run-datareader[1016]: 2021-04-15 13:43:36,252 - paho - ERROR - failed to receive on socket: [Errno 104] Connection reset by peer
Apr 15 13:43:36 datareader0899 run-datareader[1016]: ERROR:paho:failed to receive on socket: [Errno 104] Connection reset by peer
edgeHub Logs (grep-ed on the device ID):
<4> 2021-04-15 10:01:01.554 +00:00 [WRN] - Closing connection for device: MTC0899_DATAREADER, Microsoft.Azure.Devices.ProtocolGateway.ProtocolGatewayException: Channel closed.,
<6> 2021-04-15 10:01:01.554 +00:00 [INF] - Disposing MessagingServiceClient for device Id MTC0899_DATAREADER because of exception - Microsoft.Azure.Devices.ProtocolGateway.ProtocolGatewayException: Channel closed.
<6> 2021-04-15 10:01:01.554 +00:00 [INF] - Setting device proxy inactive for device Id MTC0899_DATAREADER
<6> 2021-04-15 10:01:01.554 +00:00 [INF] - Removing device connection for device MTC0899_DATAREADER with removeCloudConnection flag 'True'.
<6> 2021-04-15 10:01:01.555 +00:00 [INF] - Closing receiver in cloud proxy bae7bd3d-1935-4e57-931d-b0d767e83808 for MTC0899_DATAREADER
<6> 2021-04-15 10:01:01.555 +00:00 [INF] - Closed cloud proxy bae7bd3d-1935-4e57-931d-b0d767e83808 for MTC0899_DATAREADER
<6> 2021-04-15 10:01:01.555 +00:00 [INF] - Device connection removed for device MTC0899_DATAREADER
<6> 2021-04-15 10:01:01.556 +00:00 [INF] - Remove device connection for device MTC0899_DATAREADER
<6> 2021-04-15 10:01:02.056 +00:00 [INF] - Client MTC0899_DATAREADER in device scope authenticated locally.
<6> 2021-04-15 10:01:02.056 +00:00 [INF] - Successfully generated identity for clientId MTC0899_DATAREADER and username <omitted>.azure-devices.net/MTC0899_DATAREADER/?api-version=2019-10-01&DeviceClientType=azure-iot-device-iothub-py%2F2.5.1%283.7.5%3BLinux%20%23145-Ubuntu%20SMP%20Wed%20Mar%2024%2018%3A08%3A07%20UTC%202021%3Bx86_64%29
<6> 2021-04-15 10:01:02.056 +00:00 [INF] - ClientAuthenticated, MTC0899_DATAREADER, 2c063b83
<6> 2021-04-15 10:01:02.056 +00:00 [INF] - New device connection for device MTC0899_DATAREADER
<6> 2021-04-15 10:01:02.056 +00:00 [INF] - Client MTC0899_DATAREADER connected to edgeHub, processing existing subscriptions.
<6> 2021-04-15 10:01:02.057 +00:00 [INF] - Bind device proxy for device MTC0899_DATAREADER
<6> 2021-04-15 10:01:02.057 +00:00 [INF] - Binding message channel for device Id MTC0899_DATAREADER
<6> 2021-04-15 10:01:02.057 +00:00 [INF] - Attempting to connect to IoT Hub for client MTC0899_DATAREADER via AMQP...
<6> 2021-04-15 10:01:02.379 +00:00 [INF] - Cloud connection for MTC0899_DATAREADER is True
<6> 2021-04-15 10:01:02.379 +00:00 [INF] - Connection status for MTC0899_DATAREADER changed to ConnectionEstablished
<6> 2021-04-15 10:01:02.379 +00:00 [INF] - Client MTC0899_DATAREADER connected to cloud, processing existing subscriptions.
<6> 2021-04-15 10:01:02.379 +00:00 [INF] - Skipping MTC0899_DATAREADER for subscription processing, as it is currently being processed.
<6> 2021-04-15 10:01:02.379 +00:00 [INF] - Created cloud proxy for client MTC0899_DATAREADER via AMQP, with client operation timeout 20 seconds.
<6> 2021-04-15 10:01:02.379 +00:00 [INF] - Initialized cloud proxy 4a075233-474d-445e-ab7d-892f449b3996 for MTC0899_DATAREADER
<6> 2021-04-15 10:01:02.379 +00:00 [INF] - Created cloud connection for client MTC0899_DATAREADER
<6> 2021-04-15 10:01:03.057 +00:00 [INF] - Processing subscriptions DesiredPropertyUpdates, TwinResponse for client MTC0899_DATAREADER.
<6> 2021-04-15 10:01:03.058 +00:00 [INF] - Set subscriptions from session state for MTC0899_DATAREADER
<6> 2021-04-15 10:01:03.058 +00:00 [INF] - Processing pending subscriptions for MTC0899_DATAREADER
<6> 2021-04-15 10:01:03.265 +00:00 [INF] - Processing subscriptions DesiredPropertyUpdates, TwinResponse for client MTC0899_DATAREADER.
<6> 2021-04-15 10:01:03.266 +00:00 [INF] - Set subscriptions from session state for MTC0899_DATAREADER
<6> 2021-04-15 10:01:03.266 +00:00 [INF] - Processing pending subscriptions for MTC0899_DATAREADER
<6> 2021-04-15 10:01:07.389 +00:00 [INF] - Updated reported properties for MTC0899_DATAREADER
<6> 2021-04-15 10:59:00.996 +00:00 [INF] - Closing connection for device: MTC0899_DATAREADER, ,
<6> 2021-04-15 10:59:00.996 +00:00 [INF] - Disposing MessagingServiceClient for device Id MTC0899_DATAREADER because of exception -
<6> 2021-04-15 10:59:00.996 +00:00 [INF] - Setting device proxy inactive for device Id MTC0899_DATAREADER
<6> 2021-04-15 10:59:00.996 +00:00 [INF] - Removing device connection for device MTC0899_DATAREADER with removeCloudConnection flag 'True'.
<6> 2021-04-15 10:59:00.996 +00:00 [INF] - Closing receiver in cloud proxy 4a075233-474d-445e-ab7d-892f449b3996 for MTC0899_DATAREADER
<6> 2021-04-15 10:59:00.997 +00:00 [INF] - Closed cloud proxy 4a075233-474d-445e-ab7d-892f449b3996 for MTC0899_DATAREADER
<6> 2021-04-15 10:59:00.997 +00:00 [INF] - Device connection removed for device MTC0899_DATAREADER
<6> 2021-04-15 10:59:00.997 +00:00 [INF] - Remove device connection for device MTC0899_DATAREADER
<6> 2021-04-15 10:59:01.019 +00:00 [INF] - Client MTC0899_DATAREADER in device scope authenticated locally.
<6> 2021-04-15 10:59:01.019 +00:00 [INF] - Successfully generated identity for clientId MTC0899_DATAREADER and username <omitted>.azure-devices.net/MTC0899_DATAREADER/?api-version=2019-10-01&DeviceClientType=azure-iot-device-iothub-py%2F2.5.1%283.7.5%3BLinux%20%23145-Ubuntu%20SMP%20Wed%20Mar%2024%2018%3A08%3A07%20UTC%202021%3Bx86_64%29
<6> 2021-04-15 10:59:01.019 +00:00 [INF] - ClientAuthenticated, MTC0899_DATAREADER, 7fe86225
<6> 2021-04-15 10:59:01.019 +00:00 [INF] - New device connection for device MTC0899_DATAREADER
<6> 2021-04-15 10:59:01.019 +00:00 [INF] - Client MTC0899_DATAREADER connected to edgeHub, processing existing subscriptions.
<6> 2021-04-15 10:59:01.019 +00:00 [INF] - Bind device proxy for device MTC0899_DATAREADER
<6> 2021-04-15 10:59:01.019 +00:00 [INF] - Binding message channel for device Id MTC0899_DATAREADER
<6> 2021-04-15 10:59:01.019 +00:00 [INF] - Attempting to connect to IoT Hub for client MTC0899_DATAREADER via AMQP...
<6> 2021-04-15 10:59:01.214 +00:00 [INF] - Cloud connection for MTC0899_DATAREADER is True
<6> 2021-04-15 10:59:01.214 +00:00 [INF] - Connection status for MTC0899_DATAREADER changed to ConnectionEstablished
<6> 2021-04-15 10:59:01.214 +00:00 [INF] - Client MTC0899_DATAREADER connected to cloud, processing existing subscriptions.
<6> 2021-04-15 10:59:01.214 +00:00 [INF] - Skipping MTC0899_DATAREADER for subscription processing, as it is currently being processed.
<6> 2021-04-15 10:59:01.214 +00:00 [INF] - Created cloud proxy for client MTC0899_DATAREADER via AMQP, with client operation timeout 20 seconds.
<6> 2021-04-15 10:59:01.214 +00:00 [INF] - Initialized cloud proxy 8cd16638-97ae-4c04-b884-ebd42c71e3d5 for MTC0899_DATAREADER
<6> 2021-04-15 10:59:01.215 +00:00 [INF] - Created cloud connection for client MTC0899_DATAREADER
<6> 2021-04-15 11:57:00.999 +00:00 [INF] - Closing connection for device: MTC0899_DATAREADER, ,
<6> 2021-04-15 11:57:00.999 +00:00 [INF] - Disposing MessagingServiceClient for device Id MTC0899_DATAREADER because of exception -
<6> 2021-04-15 11:57:00.999 +00:00 [INF] - Setting device proxy inactive for device Id MTC0899_DATAREADER
<6> 2021-04-15 11:57:00.999 +00:00 [INF] - Removing device connection for device MTC0899_DATAREADER with removeCloudConnection flag 'True'.
<6> 2021-04-15 11:57:00.999 +00:00 [INF] - Closing receiver in cloud proxy 8cd16638-97ae-4c04-b884-ebd42c71e3d5 for MTC0899_DATAREADER
<6> 2021-04-15 11:57:00.999 +00:00 [INF] - Closed cloud proxy 8cd16638-97ae-4c04-b884-ebd42c71e3d5 for MTC0899_DATAREADER
<6> 2021-04-15 11:57:00.999 +00:00 [INF] - Device connection removed for device MTC0899_DATAREADER
<6> 2021-04-15 11:57:00.999 +00:00 [INF] - Remove device connection for device MTC0899_DATAREADER
<6> 2021-04-15 11:57:01.021 +00:00 [INF] - Client MTC0899_DATAREADER in device scope authenticated locally.
<6> 2021-04-15 11:57:01.022 +00:00 [INF] - Successfully generated identity for clientId MTC0899_DATAREADER and username <omitted>.azure-devices.net/MTC0899_DATAREADER/?api-version=2019-10-01&DeviceClientType=azure-iot-device-iothub-py%2F2.5.1%283.7.5%3BLinux%20%23145-Ubuntu%20SMP%20Wed%20Mar%2024%2018%3A08%3A07%20UTC%202021%3Bx86_64%29
<6> 2021-04-15 11:57:01.022 +00:00 [INF] - ClientAuthenticated, MTC0899_DATAREADER, 013a4c9f
<6> 2021-04-15 11:57:01.022 +00:00 [INF] - New device connection for device MTC0899_DATAREADER
<6> 2021-04-15 11:57:01.022 +00:00 [INF] - Client MTC0899_DATAREADER connected to edgeHub, processing existing subscriptions.
<6> 2021-04-15 11:57:01.022 +00:00 [INF] - Bind device proxy for device MTC0899_DATAREADER
<6> 2021-04-15 11:57:01.022 +00:00 [INF] - Binding message channel for device Id MTC0899_DATAREADER
<6> 2021-04-15 11:57:01.076 +00:00 [INF] - Attempting to connect to IoT Hub for client MTC0899_DATAREADER via AMQP...
<6> 2021-04-15 11:57:01.411 +00:00 [INF] - Cloud connection for MTC0899_DATAREADER is True
<6> 2021-04-15 11:57:01.411 +00:00 [INF] - Connection status for MTC0899_DATAREADER changed to ConnectionEstablished
<6> 2021-04-15 11:57:01.412 +00:00 [INF] - Client MTC0899_DATAREADER connected to cloud, processing existing subscriptions.
<6> 2021-04-15 11:57:01.412 +00:00 [INF] - Skipping MTC0899_DATAREADER for subscription processing, as it is currently being processed.
<6> 2021-04-15 11:57:01.412 +00:00 [INF] - Created cloud proxy for client MTC0899_DATAREADER via AMQP, with client operation timeout 20 seconds.
<6> 2021-04-15 11:57:01.412 +00:00 [INF] - Initialized cloud proxy a7b99289-fb29-41ef-8ef2-205eaa98daea for MTC0899_DATAREADER
<6> 2021-04-15 11:57:01.412 +00:00 [INF] - Created cloud connection for client MTC0899_DATAREADER
<4> 2021-04-15 12:01:01.713 +00:00 [WRN] - Closing connection for device: MTC0899_DATAREADER, Microsoft.Azure.Devices.ProtocolGateway.ProtocolGatewayException: Channel closed.,
<6> 2021-04-15 12:01:01.713 +00:00 [INF] - Disposing MessagingServiceClient for device Id MTC0899_DATAREADER because of exception - Microsoft.Azure.Devices.ProtocolGateway.ProtocolGatewayException: Channel closed.
<6> 2021-04-15 12:01:01.713 +00:00 [INF] - Setting device proxy inactive for device Id MTC0899_DATAREADER
<6> 2021-04-15 12:01:01.713 +00:00 [INF] - Removing device connection for device MTC0899_DATAREADER with removeCloudConnection flag 'True'.
<6> 2021-04-15 12:01:01.714 +00:00 [INF] - Closing receiver in cloud proxy a7b99289-fb29-41ef-8ef2-205eaa98daea for MTC0899_DATAREADER
<6> 2021-04-15 12:01:01.714 +00:00 [INF] - Closed cloud proxy a7b99289-fb29-41ef-8ef2-205eaa98daea for MTC0899_DATAREADER
<6> 2021-04-15 12:01:01.714 +00:00 [INF] - Device connection removed for device MTC0899_DATAREADER
<6> 2021-04-15 12:01:01.714 +00:00 [INF] - Remove device connection for device MTC0899_DATAREADER
<6> 2021-04-15 12:01:02.228 +00:00 [INF] - Client MTC0899_DATAREADER in device scope authenticated locally.
<6> 2021-04-15 12:01:02.228 +00:00 [INF] - Successfully generated identity for clientId MTC0899_DATAREADER and username <omitted>.azure-devices.net/MTC0899_DATAREADER/?api-version=2019-10-01&DeviceClientType=azure-iot-device-iothub-py%2F2.5.1%283.7.5%3BLinux%20%23145-Ubuntu%20SMP%20Wed%20Mar%2024%2018%3A08%3A07%20UTC%202021%3Bx86_64%29
<6> 2021-04-15 12:01:02.228 +00:00 [INF] - ClientAuthenticated, MTC0899_DATAREADER, 05ceafb4
<6> 2021-04-15 12:01:02.229 +00:00 [INF] - New device connection for device MTC0899_DATAREADER
<6> 2021-04-15 12:01:02.229 +00:00 [INF] - Client MTC0899_DATAREADER connected to edgeHub, processing existing subscriptions.
<6> 2021-04-15 12:01:02.229 +00:00 [INF] - Bind device proxy for device MTC0899_DATAREADER
<6> 2021-04-15 12:01:02.229 +00:00 [INF] - Binding message channel for device Id MTC0899_DATAREADER
<6> 2021-04-15 12:01:02.229 +00:00 [INF] - Attempting to connect to IoT Hub for client MTC0899_DATAREADER via AMQP...
<6> 2021-04-15 12:01:02.420 +00:00 [INF] - Cloud connection for MTC0899_DATAREADER is True
<6> 2021-04-15 12:01:02.420 +00:00 [INF] - Connection status for MTC0899_DATAREADER changed to ConnectionEstablished
<6> 2021-04-15 12:01:02.420 +00:00 [INF] - Client MTC0899_DATAREADER connected to cloud, processing existing subscriptions.
<6> 2021-04-15 12:01:02.420 +00:00 [INF] - Skipping MTC0899_DATAREADER for subscription processing, as it is currently being processed.
<6> 2021-04-15 12:01:02.420 +00:00 [INF] - Created cloud proxy for client MTC0899_DATAREADER via AMQP, with client operation timeout 20 seconds.
<6> 2021-04-15 12:01:02.420 +00:00 [INF] - Initialized cloud proxy 317fbbc2-82b6-4c1b-b3a9-7b94d16727f7 for MTC0899_DATAREADER
<6> 2021-04-15 12:01:02.420 +00:00 [INF] - Created cloud connection for client MTC0899_DATAREADER
<6> 2021-04-15 12:01:03.231 +00:00 [INF] - Processing subscriptions DesiredPropertyUpdates, TwinResponse for client MTC0899_DATAREADER.
<6> 2021-04-15 12:01:03.231 +00:00 [INF] - Set subscriptions from session state for MTC0899_DATAREADER
<6> 2021-04-15 12:01:03.231 +00:00 [INF] - Processing pending subscriptions for MTC0899_DATAREADER
<6> 2021-04-15 12:01:03.432 +00:00 [INF] - Processing subscriptions DesiredPropertyUpdates, TwinResponse for client MTC0899_DATAREADER.
<6> 2021-04-15 12:01:03.432 +00:00 [INF] - Set subscriptions from session state for MTC0899_DATAREADER
<6> 2021-04-15 12:01:03.433 +00:00 [INF] - Processing pending subscriptions for MTC0899_DATAREADER
<6> 2021-04-15 12:01:07.556 +00:00 [INF] - Updated reported properties for MTC0899_DATAREADER
<6> 2021-04-15 12:59:01.996 +00:00 [INF] - Closing connection for device: MTC0899_DATAREADER, ,
<6> 2021-04-15 12:59:01.996 +00:00 [INF] - Disposing MessagingServiceClient for device Id MTC0899_DATAREADER because of exception -
<6> 2021-04-15 12:59:01.996 +00:00 [INF] - Setting device proxy inactive for device Id MTC0899_DATAREADER
<6> 2021-04-15 12:59:01.996 +00:00 [INF] - Removing device connection for device MTC0899_DATAREADER with removeCloudConnection flag 'True'.
<6> 2021-04-15 12:59:01.997 +00:00 [INF] - Closing receiver in cloud proxy 317fbbc2-82b6-4c1b-b3a9-7b94d16727f7 for MTC0899_DATAREADER
<6> 2021-04-15 12:59:01.997 +00:00 [INF] - Closed cloud proxy 317fbbc2-82b6-4c1b-b3a9-7b94d16727f7 for MTC0899_DATAREADER
<6> 2021-04-15 12:59:01.997 +00:00 [INF] - Device connection removed for device MTC0899_DATAREADER
<6> 2021-04-15 12:59:01.997 +00:00 [INF] - Remove device connection for device MTC0899_DATAREADER
<6> 2021-04-15 12:59:02.013 +00:00 [INF] - Client MTC0899_DATAREADER in device scope authenticated locally.
<6> 2021-04-15 12:59:02.013 +00:00 [INF] - Successfully generated identity for clientId MTC0899_DATAREADER and username <omitted>.azure-devices.net/MTC0899_DATAREADER/?api-version=2019-10-01&DeviceClientType=azure-iot-device-iothub-py%2F2.5.1%283.7.5%3BLinux%20%23145-Ubuntu%20SMP%20Wed%20Mar%2024%2018%3A08%3A07%20UTC%202021%3Bx86_64%29
<6> 2021-04-15 12:59:02.013 +00:00 [INF] - ClientAuthenticated, MTC0899_DATAREADER, 441fe13f
<6> 2021-04-15 12:59:02.013 +00:00 [INF] - New device connection for device MTC0899_DATAREADER
<6> 2021-04-15 12:59:02.014 +00:00 [INF] - Client MTC0899_DATAREADER connected to edgeHub, processing existing subscriptions.
<6> 2021-04-15 12:59:02.014 +00:00 [INF] - Bind device proxy for device MTC0899_DATAREADER
<6> 2021-04-15 12:59:02.014 +00:00 [INF] - Binding message channel for device Id MTC0899_DATAREADER
<6> 2021-04-15 12:59:02.014 +00:00 [INF] - Attempting to connect to IoT Hub for client MTC0899_DATAREADER via AMQP...
<6> 2021-04-15 12:59:02.553 +00:00 [INF] - Cloud connection for MTC0899_DATAREADER is True
<6> 2021-04-15 12:59:02.553 +00:00 [INF] - Connection status for MTC0899_DATAREADER changed to ConnectionEstablished
<6> 2021-04-15 12:59:02.553 +00:00 [INF] - Client MTC0899_DATAREADER connected to cloud, processing existing subscriptions.
<6> 2021-04-15 12:59:02.553 +00:00 [INF] - Skipping MTC0899_DATAREADER for subscription processing, as it is currently being processed.
<6> 2021-04-15 12:59:02.553 +00:00 [INF] - Created cloud proxy for client MTC0899_DATAREADER via AMQP, with client operation timeout 20 seconds.
<6> 2021-04-15 12:59:02.553 +00:00 [INF] - Initialized cloud proxy daa34d57-b09e-4eb9-b04a-e24dec683d12 for MTC0899_DATAREADER
<6> 2021-04-15 12:59:02.553 +00:00 [INF] - Created cloud connection for client MTC0899_DATAREADER
<4> 2021-04-15 13:39:05.247 +00:00 [WRN] - Closing connection for device: MTC0899_DATAREADER, Microsoft.Azure.Devices.ProtocolGateway.ProtocolGatewayException: Channel closed.,
<6> 2021-04-15 13:39:05.248 +00:00 [INF] - Disposing MessagingServiceClient for device Id MTC0899_DATAREADER because of exception - Microsoft.Azure.Devices.ProtocolGateway.ProtocolGatewayException: Channel closed.
<6> 2021-04-15 13:39:05.248 +00:00 [INF] - Setting device proxy inactive for device Id MTC0899_DATAREADER
<6> 2021-04-15 13:39:05.248 +00:00 [INF] - Removing device connection for device MTC0899_DATAREADER with removeCloudConnection flag 'True'.
<6> 2021-04-15 13:39:05.249 +00:00 [INF] - Closing receiver in cloud proxy daa34d57-b09e-4eb9-b04a-e24dec683d12 for MTC0899_DATAREADER
<6> 2021-04-15 13:39:05.249 +00:00 [INF] - Closed cloud proxy daa34d57-b09e-4eb9-b04a-e24dec683d12 for MTC0899_DATAREADER
<6> 2021-04-15 13:39:05.249 +00:00 [INF] - Device connection removed for device MTC0899_DATAREADER
<6> 2021-04-15 13:39:05.249 +00:00 [INF] - Remove device connection for device MTC0899_DATAREADER
<6> 2021-04-15 13:39:18.027 +00:00 [INF] - Client MTC0899_DATAREADER in device scope authenticated locally.
<6> 2021-04-15 13:39:18.027 +00:00 [INF] - Successfully generated identity for clientId MTC0899_DATAREADER and username <omitted>.azure-devices.net/MTC0899_DATAREADER/?api-version=2019-10-01&DeviceClientType=azure-iot-device-iothub-py%2F2.5.1%283.7.5%3BLinux%20%23145-Ubuntu%20SMP%20Wed%20Mar%2024%2018%3A08%3A07%20UTC%202021%3Bx86_64%29
<6> 2021-04-15 13:39:18.027 +00:00 [INF] - ClientAuthenticated, MTC0899_DATAREADER, 087de596
<6> 2021-04-15 13:39:18.027 +00:00 [INF] - New device connection for device MTC0899_DATAREADER
<6> 2021-04-15 13:39:18.028 +00:00 [INF] - Client MTC0899_DATAREADER connected to edgeHub, processing existing subscriptions.
<6> 2021-04-15 13:39:18.028 +00:00 [INF] - Bind device proxy for device MTC0899_DATAREADER
<6> 2021-04-15 13:39:18.028 +00:00 [INF] - Binding message channel for device Id MTC0899_DATAREADER
<6> 2021-04-15 13:39:18.028 +00:00 [INF] - Attempting to connect to IoT Hub for client MTC0899_DATAREADER via AMQP...
<6> 2021-04-15 13:39:18.215 +00:00 [INF] - Cloud connection for MTC0899_DATAREADER is True
<6> 2021-04-15 13:39:18.215 +00:00 [INF] - Connection status for MTC0899_DATAREADER changed to ConnectionEstablished
<6> 2021-04-15 13:39:18.215 +00:00 [INF] - Client MTC0899_DATAREADER connected to cloud, processing existing subscriptions.
<6> 2021-04-15 13:39:18.216 +00:00 [INF] - Skipping MTC0899_DATAREADER for subscription processing, as it is currently being processed.
<6> 2021-04-15 13:39:18.216 +00:00 [INF] - Created cloud proxy for client MTC0899_DATAREADER via AMQP, with client operation timeout 20 seconds.
<6> 2021-04-15 13:39:18.216 +00:00 [INF] - Initialized cloud proxy ef3c2e34-f076-4350-9cd4-8bdfc765fed6 for MTC0899_DATAREADER
<6> 2021-04-15 13:39:18.216 +00:00 [INF] - Created cloud connection for client MTC0899_DATAREADER
<6> 2021-04-15 13:39:19.031 +00:00 [INF] - Processing subscriptions DesiredPropertyUpdates, TwinResponse for client MTC0899_DATAREADER.
<6> 2021-04-15 13:39:19.031 +00:00 [INF] - Set subscriptions from session state for MTC0899_DATAREADER
<6> 2021-04-15 13:39:19.031 +00:00 [INF] - Processing pending subscriptions for MTC0899_DATAREADER
<6> 2021-04-15 13:39:19.230 +00:00 [INF] - Processing subscriptions DesiredPropertyUpdates, TwinResponse for client MTC0899_DATAREADER.
<6> 2021-04-15 13:39:19.230 +00:00 [INF] - Set subscriptions from session state for MTC0899_DATAREADER
<6> 2021-04-15 13:39:19.230 +00:00 [INF] - Processing pending subscriptions for MTC0899_DATAREADER
<6> 2021-04-15 13:39:23.593 +00:00 [INF] - Updated reported properties for MTC0899_DATAREADER
<6> 2021-04-15 13:44:08.147 +00:00 [INF] - Client MTC0899_DATAREADER in device scope authenticated locally.
<6> 2021-04-15 13:44:08.147 +00:00 [INF] - Successfully generated identity for clientId MTC0899_DATAREADER and username <omitted>.azure-devices.net/MTC0899_DATAREADER/?api-version=2019-10-01&DeviceClientType=azure-iot-device-iothub-py%2F2.5.1%283.7.5%3BLinux%20%23145-Ubuntu%20SMP%20Wed%20Mar%2024%2018%3A08%3A07%20UTC%202021%3Bx86_64%29
<6> 2021-04-15 13:44:08.147 +00:00 [INF] - ClientAuthenticated, MTC0899_DATAREADER, 45383948
<6> 2021-04-15 13:44:08.148 +00:00 [INF] - New device connection for device MTC0899_DATAREADER
<6> 2021-04-15 13:44:08.148 +00:00 [INF] - Client MTC0899_DATAREADER connected to edgeHub, processing existing subscriptions.
<6> 2021-04-15 13:44:08.149 +00:00 [INF] - Bind device proxy for device MTC0899_DATAREADER
<6> 2021-04-15 13:44:08.149 +00:00 [INF] - Binding message channel for device Id MTC0899_DATAREADER
<6> 2021-04-15 13:44:08.149 +00:00 [INF] - Attempting to connect to IoT Hub for client MTC0899_DATAREADER via AMQP...
<6> 2021-04-15 13:44:08.604 +00:00 [INF] - Cloud connection for MTC0899_DATAREADER is True
<6> 2021-04-15 13:44:08.604 +00:00 [INF] - Connection status for MTC0899_DATAREADER changed to ConnectionEstablished
<6> 2021-04-15 13:44:08.604 +00:00 [INF] - Client MTC0899_DATAREADER connected to cloud, processing existing subscriptions.
<6> 2021-04-15 13:44:08.604 +00:00 [INF] - Skipping MTC0899_DATAREADER for subscription processing, as it is currently being processed.
<6> 2021-04-15 13:44:08.604 +00:00 [INF] - Created cloud proxy for client MTC0899_DATAREADER via AMQP, with client operation timeout 20 seconds.
<6> 2021-04-15 13:44:08.604 +00:00 [INF] - Initialized cloud proxy 5202ec5b-da2d-4c1c-9658-e5cc9f81b82d for MTC0899_DATAREADER
<6> 2021-04-15 13:44:08.604 +00:00 [INF] - Created cloud connection for client MTC0899_DATAREADER
ASA on IoT Edge logs (to show when event stream stopped):
{"log":"\n","stream":"stdout","time":"2021-04-15T10:26:20.81528111Z"}
{"log":"2021-04-15 10:27:20.819 +00:00 [INF] - \n","stream":"stdout","time":"2021-04-15T10:27:20.819497136Z"}
{"log":"=============================Metrics report=============================\n","stream":"stdout","time":"2021-04-15T10:27:20.819519932Z"}
{"log":" ConversionErrors : 0\n","stream":"stdout","time":"2021-04-15T10:27:20.819523273Z"}
{"log":" InputEventBytes : 15.64 MiB\n","stream":"stdout","time":"2021-04-15T10:27:20.819526054Z"}
{"log":" Errors : 0\n","stream":"stdout","time":"2021-04-15T10:27:20.81954704Z"}
{"log":" EarlyInputEvents : 0\n","stream":"stdout","time":"2021-04-15T10:27:20.819565577Z"}
{"log":" InputEvents : 34.45 K\n","stream":"stdout","time":"2021-04-15T10:27:20.819569004Z"}
{"log":" OutputEvents : 330\n","stream":"stdout","time":"2021-04-15T10:27:20.819571764Z"}
{"log":" InputEventsSources : 34.45 K\n","stream":"stdout","time":"2021-04-15T10:27:20.819574659Z"}
{"log":" LateInputEvents : 0\n","stream":"stdout","time":"2021-04-15T10:27:20.81957756Z"}
{"log":" DeserializationError : 0\n","stream":"stdout","time":"2021-04-15T10:27:20.819580287Z"}
{"log":" DroppedOrAdjustedEvents : 0\n","stream":"stdout","time":"2021-04-15T10:27:20.819582908Z"}
{"log":" OutputWatermarkDelaySeconds : 0\n","stream":"stdout","time":"2021-04-15T10:27:20.819585543Z"}
{"log":"========================================================================\n","stream":"stdout","time":"2021-04-15T10:27:20.819588461Z"}
{"log":"\n","stream":"stdout","time":"2021-04-15T10:27:20.819591231Z"}
{"log":"2021-04-15 10:28:20.824 +00:00 [INF] - \n","stream":"stdout","time":"2021-04-15T10:28:20.824717368Z"}
{"log":"=============================Metrics report=============================\n","stream":"stdout","time":"2021-04-15T10:28:20.82474323Z"}
{"log":" ConversionErrors : 0\n","stream":"stdout","time":"2021-04-15T10:28:20.824755576Z"}
{"log":" InputEventBytes : 15.45 MiB\n","stream":"stdout","time":"2021-04-15T10:28:20.824758134Z"}
{"log":" Errors : 0\n","stream":"stdout","time":"2021-04-15T10:28:20.824760549Z"}
{"log":" EarlyInputEvents : 0\n","stream":"stdout","time":"2021-04-15T10:28:20.824762859Z"}
{"log":" InputEvents : 33.61 K\n","stream":"stdout","time":"2021-04-15T10:28:20.82476519Z"}
{"log":" OutputEvents : 330\n","stream":"stdout","time":"2021-04-15T10:28:20.824767699Z"}
{"log":" InputEventsSources : 33.61 K\n","stream":"stdout","time":"2021-04-15T10:28:20.82477001Z"}
{"log":" LateInputEvents : 0\n","stream":"stdout","time":"2021-04-15T10:28:20.824772373Z"}
{"log":" DeserializationError : 0\n","stream":"stdout","time":"2021-04-15T10:28:20.824774727Z"}
{"log":" DroppedOrAdjustedEvents : 0\n","stream":"stdout","time":"2021-04-15T10:28:20.824777006Z"}
{"log":" OutputWatermarkDelaySeconds : 0\n","stream":"stdout","time":"2021-04-15T10:28:20.824779273Z"}
{"log":"========================================================================\n","stream":"stdout","time":"2021-04-15T10:28:20.824781565Z"}
{"log":"\n","stream":"stdout","time":"2021-04-15T10:28:20.824783942Z"}
{"log":"2021-04-15 10:29:20.824 +00:00 [INF] - \n","stream":"stdout","time":"2021-04-15T10:29:20.824228586Z"}
{"log":"=============================Metrics report=============================\n","stream":"stdout","time":"2021-04-15T10:29:20.824251098Z"}
{"log":" ConversionErrors : 0\n","stream":"stdout","time":"2021-04-15T10:29:20.824254746Z"}
{"log":" InputEventBytes : 1.90 MiB\n","stream":"stdout","time":"2021-04-15T10:29:20.824257504Z"}
{"log":" Errors : 0\n","stream":"stdout","time":"2021-04-15T10:29:20.824260228Z"}
{"log":" EarlyInputEvents : 0\n","stream":"stdout","time":"2021-04-15T10:29:20.82426291Z"}
{"log":" InputEvents : 4.14 K\n","stream":"stdout","time":"2021-04-15T10:29:20.82426568Z"}
{"log":" OutputEvents : 41\n","stream":"stdout","time":"2021-04-15T10:29:20.824268326Z"}
{"log":" InputEventsSources : 4.14 K\n","stream":"stdout","time":"2021-04-15T10:29:20.824270912Z"}
{"log":" LateInputEvents : 0\n","stream":"stdout","time":"2021-04-15T10:29:20.824273504Z"}
{"log":" DeserializationError : 0\n","stream":"stdout","time":"2021-04-15T10:29:20.824276435Z"}
{"log":" DroppedOrAdjustedEvents : 0\n","stream":"stdout","time":"2021-04-15T10:29:20.824279125Z"}
{"log":" OutputWatermarkDelaySeconds : 11.3419822\n","stream":"stdout","time":"2021-04-15T10:29:20.82428179Z"}
{"log":"========================================================================\n","stream":"stdout","time":"2021-04-15T10:29:20.824284481Z"}
{"log":"\n","stream":"stdout","time":"2021-04-15T10:29:20.824287186Z"}
{"log":"2021-04-15 10:30:20.823 +00:00 [INF] - \n","stream":"stdout","time":"2021-04-15T10:30:20.823993928Z"}
{"log":"=============================Metrics report=============================\n","stream":"stdout","time":"2021-04-15T10:30:20.824018018Z"}
{"log":" ConversionErrors : 0\n","stream":"stdout","time":"2021-04-15T10:30:20.824021757Z"}
{"log":" InputEventBytes : 0 B\n","stream":"stdout","time":"2021-04-15T10:30:20.824032597Z"}
{"log":" Errors : 0\n","stream":"stdout","time":"2021-04-15T10:30:20.824035152Z"}
{"log":" EarlyInputEvents : 0\n","stream":"stdout","time":"2021-04-15T10:30:20.824037473Z"}
{"log":" InputEvents : 0\n","stream":"stdout","time":"2021-04-15T10:30:20.824039787Z"}
{"log":" OutputEvents : 30\n","stream":"stdout","time":"2021-04-15T10:30:20.824042164Z"}
{"log":" InputEventsSources : 0\n","stream":"stdout","time":"2021-04-15T10:30:20.824044471Z"}
{"log":" LateInputEvents : 0\n","stream":"stdout","time":"2021-04-15T10:30:20.824046959Z"}
{"log":" DeserializationError : 0\n","stream":"stdout","time":"2021-04-15T10:30:20.824049382Z"}
{"log":" DroppedOrAdjustedEvents : 0\n","stream":"stdout","time":"2021-04-15T10:30:20.824051733Z"}
{"log":" OutputWatermarkDelaySeconds : 11.340563\n","stream":"stdout","time":"2021-04-15T10:30:20.824054054Z"}
{"log":"========================================================================\n","stream":"stdout","time":"2021-04-15T10:30:20.824056359Z"}
{"log":"\n","stream":"stdout","time":"2021-04-15T10:30:20.824058745Z"}
{"log":"2021-04-15 10:31:20.828 +00:00 [INF] - \n","stream":"stdout","time":"2021-04-15T10:31:20.828375363Z"}
{"log":"=============================Metrics report=============================\n","stream":"stdout","time":"2021-04-15T10:31:20.828396272Z"}
{"log":" ConversionErrors : 0\n","stream":"stdout","time":"2021-04-15T10:31:20.828399882Z"}
{"log":" InputEventBytes : 0 B\n","stream":"stdout","time":"2021-04-15T10:31:20.828402619Z"}
{"log":" Errors : 0\n","stream":"stdout","time":"2021-04-15T10:31:20.828405826Z"}
{"log":" EarlyInputEvents : 0\n","stream":"stdout","time":"2021-04-15T10:31:20.828408533Z"}
{"log":" InputEvents : 0\n","stream":"stdout","time":"2021-04-15T10:31:20.828411181Z"}
{"log":" OutputEvents : 0\n","stream":"stdout","time":"2021-04-15T10:31:20.828413797Z"}
{"log":" InputEventsSources : 0\n","stream":"stdout","time":"2021-04-15T10:31:20.828416411Z"}
{"log":" LateInputEvents : 0\n","stream":"stdout","time":"2021-04-15T10:31:20.828419025Z"}
{"log":" DeserializationError : 0\n","stream":"stdout","time":"2021-04-15T10:31:20.828421848Z"}
{"log":" DroppedOrAdjustedEvents : 0\n","stream":"stdout","time":"2021-04-15T10:31:20.828424576Z"}
{"log":" OutputWatermarkDelaySeconds : 11.3403962\n","stream":"stdout","time":"2021-04-15T10:31:20.828427268Z"}
{"log":"========================================================================\n","stream":"stdout","time":"2021-04-15T10:31:20.828429968Z"}
Issue Analytics
- State:
- Created 2 years ago
- Comments:5 (2 by maintainers)
Top Results From Across the Web
Client disconnects - and does not reconnect back , the caller ...
ConnectionDroppedError : Unexpected disconnection\n'] ... from azure.iot.device.aio import IoTHubDeviceClient ##creating client
Read more >azure.iot.device.aio.IoTHubDeviceClient class | Microsoft Learn
Receive a message that has been sent from the Azure IoT Hub. If no message is yet available, will wait until an item...
Read more >python - Why do I keep getting module 'azure' not found error ...
Your post doesn't say how your environment is managed, where it is and such, but this is a problem I have often when...
Read more >A Python SDK for connecting devices to Microsoft Azure IoT ...
feature available ✖️ feature planned but not yet supported ➖ no support ... from azure.iot.device.aio import IoTHubDeviceClient from ...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
@jackt-moran
I’m going to close this issue now since you haven’t been able to repro it. If it shows up again, don’t hesitate to open another GitHub issue, and we’ll look into it
@cartertinney no problem about the delay, I appreciate you getting back to me, especially given that I don’t have repro steps handy.
To the points you listed above:
.shutdown()in afinallyblock of our top levelrun()coroutine. In theory it is possible.shutdown()never gets called on occasion if one of the previous steps in thefinallyblock hangs. Systemd will kill a process that doesn’t shutdown gracefully within some amount of time such as 60 seconds. In practice, I haven’t observed this happen, but I will also verify the behavior and modify the code to guarantee.shutdown()is always called.It’s worth noting that I haven’t seen this issue reproduce itself since I first created this issue, and since I don’t know what caused it in the first place I can’t reliably reproduce myself. This will, in particular, make (3) a bit hard, but I will update this issue with anything that I find.