Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
IotHub call timed out - direct methods stop working, D2C messages unaffected
See original GitHub issueHi,
This appears related to #534 but I am raising it as a new ticket as I have now collected some good DBG logs.
Expected Behavior
Direct Method calls can be made to a module reliably
Current Behavior
Direct Method calls start failing after some hours of running.
This happened on 3 remote sites simultaneously, there is no shared network between the sites (they are connected to the iothub via 3G routers). Direct methods continued working on a further 3 sites at that time.
D2C messages continue to work throughout for all devices (6 in total)
Restarting the affected module does not fix the issue. Restarting the edgeHub module does not fix the issue. Restarting the iotedge service via systemctl does fix the issue.
The point at which they stop corresponds with a time out entry in the edgeHub logs (DBG only). This was verified by polling the direct methods every 30 seconds and looking in the logs at the exact point they stopped.
Steps to Reproduce
Create a module with a direct method (module is based on the sample Temperature polling module).
Leave running, after hours / days, the direct method calls will return ‘Timed out waiting for device to connect’. Connectivity will never be restored until the iotedge service itself is restarted
Context (Environment)
Ubuntu 18.04, amd64, linux containers
Runtime Versions
iotedged
iotedge 1.0.4 (2835ac47fbb944ba628a5cfe812ff394768842e0)
Edge Agent
1.0.0
Edge Hub
1.0.0
Docker
3.0.1
SDK and protocol of affected module
Microsoft.Azure.Devices.Client 1.18.1 Transport Type is Amqp_Tcp_Only
Logs
Here are the logs for the edgeHub of the first affected device. The last successful direct method call was
at 01:24:00 exactly. 30 seconds later, another call was made and failed.
The first line is the last successful call and then 18 seconds later we get a connection error. The logs then show a successful reconnection. D2C messages carry on working, but no further Direct Method calls can be made.
2018-12-13 01:24:00.357 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudProxy] - Received call method from cloud for device dellgw5000-8/modbus
2018-12-13 01:24:00.358 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Routing.RoutingEdgeHub] - Received method invoke call from dellgw5000-8/modbus for dellgw5000-8/modbus with correlation ID 09bd6727-9ed7-4c18-9f5e-96e15de78c80
2018-12-13 01:24:00.358 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.InvokeMethodHandler] - Invoking method start on client dellgw5000-8/modbus.
2018-12-13 01:24:00.360 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudProxy] - Received call method from cloud for device dellgw5000-8/modbus
2018-12-13 01:24:00.360 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Device.DeviceMessageHandler] - Sent method invoke call from device/module dellgw5000-8/modbus for dellgw5000-8/modbus with correlation ID 09bd6727-9ed7-4c18-9f5e-96e15de78c80
2018-12-13 01:24:00.360 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Routing.RoutingEdgeHub] - Received method invoke call from dellgw5000-8/modbus for dellgw5000-8/modbus with correlation ID 77d6160e-bfb6-43df-a9e1-0eb12923258b
2018-12-13 01:24:00.360 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.InvokeMethodHandler] - Invoking method start on client dellgw5000-8/modbus.
2018-12-13 01:24:00.396 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Device.DeviceMessageHandler] - Sent method invoke call from device/module dellgw5000-8/modbus for dellgw5000-8/modbus with correlation ID 77d6160e-bfb6-43df-a9e1-0eb12923258b
2018-12-13 01:24:00.498 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Device.DeviceMessageHandler] - Received response for method invoke call with correlation ID 09bd6727-9ed7-4c18-9f5e-96e15de78c80
2018-12-13 01:24:00.638 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Device.DeviceMessageHandler] - Received response for method invoke call with correlation ID 77d6160e-bfb6-43df-a9e1-0eb12923258b
2018-12-13 01:24:03.382 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Amqp.LinkHandlers.EventsLinkHandler] - EventsLinkHandler processed 1 messages for dellgw5000-8/modbus
2018-12-13 01:24:11.304 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Amqp.LinkHandlers.EventsLinkHandler] - EventsLinkHandler processed 1 messages for dellgw5000-8/modbus
2018-12-13 01:24:18.385 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient] - Operation SendEventAsync timed out for dellgw5000-8/modbus
2018-12-13 01:24:18.390 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceConnectivityManager] - IotHub call timed out
2018-12-13 01:24:18.393 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceConnectivityManager] - Exiting connected state
2018-12-13 01:24:18.396 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceConnectivityManager] - Entering unreachable state
2018-12-13 01:24:18.407 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceConnectivityManager] - Calling IotHub to test connectivity
2018-12-13 01:24:18.424 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.ConnectionManager] - Obtained cloud connection for device dellgw5000-8/$edgeHub
2018-12-13 01:24:18.542 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient] - Received connection status changed callback with connection status Disabled and reason Client_Close for dellgw5000-8/modbus
2018-12-13 01:24:18.452 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudProxy] - Error sending message for device dellgw5000-8/modbus in cloud proxy 67010973-120e-462c-b3c3-b30f8cc63911
System.TimeoutException: Operation timeout expired.
at Microsoft.Azure.Devices.Client.InternalClient.<>c.<ApplyTimeout>b__62_2(Task t)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass28_0.<<InvokeFunc>b__0>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 172
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 132
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudProxy.SendMessageAsync(IMessage inputMessage) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudProxy.cs:line 127
2018-12-13 01:24:18.606 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Routing.CloudEndpoint] - Retrying sending message from dellgw5000-8/modbus to Iot Hub due to exception System.TimeoutException:Operation timeout expired..
2018-12-13 01:24:18.695 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceConnectivityManager] - IotHub call succeeded
2018-12-13 01:24:18.695 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceConnectivityManager] - Entering connected state
2018-12-13 01:24:18.696 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient] - Operation UpdateReportedPropertiesAsync succeeded for dellgw5000-8/$edgeHub
2018-12-13 01:24:18.696 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudProxy] - Updating reported properties for device dellgw5000-8/$edgeHub
2018-12-13 01:24:19.647 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Routing.CloudEndpoint] - Sending 1 message(s) upstream.
2018-12-13 01:24:19.648 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.ConnectionManager] - Obtained cloud connection for device dellgw5000-8/modbus
2018-12-13 01:24:20.855 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Amqp.LinkHandlers.EventsLinkHandler] - EventsLinkHandler processed 1 messages for dellgw5000-8/modbus
2018-12-13 01:24:20.856 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceConnectivityManager] - IotHub call succeeded
2018-12-13 01:24:20.856 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient] - Operation SendEventAsync succeeded for dellgw5000-8/modbus
2018-12-13 01:24:20.857 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudProxy] - Sending message for device dellgw5000-8/modbus
2018-12-13 01:24:20.858 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Storage.MessageStore] - Getting next batch for endpoint iothub starting from 57971 with batch size 10.
2018-12-13 01:24:20.859 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Storage.MessageStore] - Obtained next batch for endpoint iothub with batch size 3. Next start offset = 57974.
2018-12-13 01:24:20.871 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Routing.CloudEndpoint] - Sending 3 message(s) upstream.
2018-12-13 01:24:20.872 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.ConnectionManager] - Obtained cloud connection for device dellgw5000-8/modbus
2018-12-13 01:24:20.929 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceConnectivityManager] - IotHub call succeeded
2018-12-13 01:24:20.930 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient] - Operation SendEventAsync succeeded for dellgw5000-8/modbus
2018-12-13 01:24:20.930 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudProxy] - Sending message for device dellgw5000-8/modbus
2018-12-13 01:24:20.932 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.ConnectionManager] - Obtained cloud connection for device dellgw5000-8/modbus
2018-12-13 01:24:20.992 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceConnectivityManager] - IotHub call succeeded
2018-12-13 01:24:20.992 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient] - Operation SendEventAsync succeeded for dellgw5000-8/modbus
2018-12-13 01:24:20.994 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudProxy] - Sending message for device dellgw5000-8/modbus
2018-12-13 01:24:20.996 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.ConnectionManager] - Obtained cloud connection for device dellgw5000-8/modbus
Here are the edgehub logs for the second affected module, again last successful call was at 01:24:00 exactly, and this module is in a different location connecting to the iothub through a completely separate 3G network. The time stamps are almost identical so it seems this must relate to the IoT hub the devices connect to.
2018-12-13 01:24:00.357 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudProxy] - Received call method from cloud for device dellgw5000-5/modbus
2018-12-13 01:24:00.358 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Routing.RoutingEdgeHub] - Received method invoke call from dellgw5000-5/modbus for dellgw5000-5/modbus with correlation ID 827009c5-13fd-4f1e-ae72-7314d0cdf9f0
2018-12-13 01:24:00.358 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.InvokeMethodHandler] - Invoking method start on client dellgw5000-5/modbus.
2018-12-13 01:24:00.361 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Device.DeviceMessageHandler] - Sent method invoke call from device/module dellgw5000-5/modbus for dellgw5000-5/modbus with correlation ID 827009c5-13fd-4f1e-ae72-7314d0cdf9f0
2018-12-13 01:24:00.499 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Device.DeviceMessageHandler] - Received response for method invoke call with correlation ID 827009c5-13fd-4f1e-ae72-7314d0cdf9f0
2018-12-13 01:24:03.382 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Amqp.LinkHandlers.EventsLinkHandler] - EventsLinkHandler processed 1 messages for dellgw5000-5/modbus
2018-12-13 01:24:08.409 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Amqp.LinkHandlers.EventsLinkHandler] - EventsLinkHandler processed 1 messages for dellgw5000-5/modbus
2018-12-13 01:24:13.443 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Amqp.LinkHandlers.EventsLinkHandler] - EventsLinkHandler processed 1 messages for dellgw5000-5/modbus
2018-12-13 01:24:18.341 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient] - Operation SendEventAsync timed out for dellgw5000-5/modbus
2018-12-13 01:24:18.344 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceConnectivityManager] - IotHub call timed out
2018-12-13 01:24:18.348 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceConnectivityManager] - Exiting connected state
2018-12-13 01:24:18.351 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceConnectivityManager] - Entering unreachable state
2018-12-13 01:24:18.363 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceConnectivityManager] - Calling IotHub to test connectivity
2018-12-13 01:24:18.380 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.ConnectionManager] - Obtained cloud connection for device dellgw5000-5/$edgeHub
2018-12-13 01:24:18.413 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient] - Received connection status changed callback with connection status Disabled and reason Client_Close for dellgw5000-5/modbus
2018-12-13 01:24:18.508 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Amqp.LinkHandlers.EventsLinkHandler] - EventsLinkHandler processed 1 messages for dellgw5000-5/modbus
2018-12-13 01:24:18.399 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudProxy] - Error sending message for device dellgw5000-5/modbus in cloud proxy f31de7ba-046b-49b6-b702-d435ddfe1a23
System.TimeoutException: Operation timeout expired.
at Microsoft.Azure.Devices.Client.InternalClient.<>c.<ApplyTimeout>b__62_2(Task t)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass28_0.<<InvokeFunc>b__0>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 172
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 132
at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudProxy.SendMessageAsync(IMessage inputMessage) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudProxy.cs:line 127
2018-12-13 01:24:18.538 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Routing.CloudEndpoint] - Retrying sending message from dellgw5000-5/modbus to Iot Hub due to exception System.TimeoutException:Operation timeout expired..
2018-12-13 01:24:18.688 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceConnectivityManager] - IotHub call succeeded
2018-12-13 01:24:18.689 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceConnectivityManager] - Entering connected state
2018-12-13 01:24:18.689 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient] - Operation UpdateReportedPropertiesAsync succeeded for dellgw5000-5/$edgeHub
2018-12-13 01:24:18.690 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudProxy] - Updating reported properties for device dellgw5000-5/$edgeHub
2018-12-13 01:24:19.581 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Routing.CloudEndpoint] - Sending 1 message(s) upstream.
2018-12-13 01:24:19.581 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.ConnectionManager] - Obtained cloud connection for device dellgw5000-5/modbus
2018-12-13 01:24:20.900 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceConnectivityManager] - IotHub call succeeded
2018-12-13 01:24:20.900 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient] - Operation SendEventAsync succeeded for dellgw5000-5/modbus
2018-12-13 01:24:20.901 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudProxy] - Sending message for device dellgw5000-5/modbus
2018-12-13 01:24:20.904 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Storage.MessageStore] - Getting next batch for endpoint iothub starting from 9917 with batch size 10.
2018-12-13 01:24:20.937 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Storage.MessageStore] - Obtained next batch for endpoint iothub with batch size 4. Next start offset = 9921.
2018-12-13 01:24:20.950 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Routing.CloudEndpoint] - Sending 4 message(s) upstream.
2018-12-13 01:24:20.951 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.ConnectionManager] - Obtained cloud connection for device dellgw5000-5/modbus
2018-12-13 01:24:21.013 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceConnectivityManager] - IotHub call succeeded
2018-12-13 01:24:21.014 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient] - Operation SendEventAsync succeeded for dellgw5000-5/modbus
2018-12-13 01:24:21.014 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudProxy] - Sending message for device dellgw5000-5/modbus
2018-12-13 01:24:21.015 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.ConnectionManager] - Obtained cloud connection for device dellgw5000-5/modbus
2018-12-13 01:24:21.083 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceConnectivityManager] - IotHub call succeeded
2018-12-13 01:24:21.084 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient] - Operation SendEventAsync succeeded for dellgw5000-5/modbus
2018-12-13 01:24:21.085 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudProxy] - Sending message for device dellgw5000-5/modbus
2018-12-13 01:24:21.086 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.ConnectionManager] - Obtained cloud connection for device dellgw5000-5/modbus
2018-12-13 01:24:21.151 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceConnectivityManager] - IotHub call succeeded
2018-12-13 01:24:21.154 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient] - Operation SendEventAsync succeeded for dellgw5000-5/modbus
2018-12-13 01:24:21.155 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudProxy] - Sending message for device dellgw5000-5/modbus
2018-12-13 01:24:21.155 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.ConnectionManager] - Obtained cloud connection for device dellgw5000-5/modbus
2018-12-13 01:24:21.223 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceConnectivityManager] - IotHub call succeeded
2018-12-13 01:24:21.224 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient] - Operation SendEventAsync succeeded for dellgw5000-5/modbus
2018-12-13 01:24:21.224 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudProxy] - Sending message for device dellgw5000-5/modbus
2018-12-13 01:24:21.228 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Storage.MessageStore] - Getting next batch for endpoint iothub starting from 9921 with batch size 10.
2018-12-13 01:24:21.228 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Hub.Core.Storage.MessageStore] - Obtained next batch for endpoint iothub with batch size 0. Next start offset = 9921.
Additional Information
This is the message you get trying to call the direct method, you get the same error trying to invoke the method through the portal. This response is more or less instant.
Unhandled Exception: System.AggregateException: One or more errors occurred. (Device {"Message":"{\"errorCode\":404103,\"trackingId\":\"cb4fbbab938e44f5a66c59d823f4304b-G:5-TimeStamp:12/13/2018 13:11:23\",\"message\":\"Timed out waiting for device to connect.\",\"info\":{\"timeout\":\"00:00:00\"},\"timestampUtc\":\"2018-12-13T13:11:23.2129387Z\"}","ExceptionMessage":""} not registered) ---> Microsoft.Azure.Devices.Common.Exceptions.DeviceNotFoundException: Device {"Message":"{\"errorCode\":404103,\"trackingId\":\"cb4fbbab938e44f5a66c59d823f4304b-G:5-TimeStamp:12/13/2018 13:11:23\",\"message\":\"Timed out waiting for device to connect.\",\"info\":{\"timeout\":\"00:00:00\"},\"timestampUtc\":\"2018-12-13T13:11:23.2129387Z\"}","ExceptionMessage":""} not registered
I get the same message on a device running in my office if I disconnect it from the network. However, direct methods always start working again once I plug the cable back in.
If there is anything you would like me to test, I am happy to oblige. Also, any possible workarounds would be very helpful as we have a customer who is keen to go live as soon as possible with this.
Issue Analytics
- State:
- Created 5 years ago
- Comments:25 (11 by maintainers)
Top Results From Across the Web
Understand Azure IoT Hub direct methods
This article describes how to use direct methods to invoke code on your devices from a service app.
Read more >Azure IoT hub Direct Method connect timeout
This error message confused me since I expect my device receive the method payload through MQTT from topic "$iothub/methods/POST/#" but I ...
Read more >nRF Connect SDK v1.8.0 Release Notes
Added support for full modem firmware update. Increased the NB-IoT time (in seconds) before the registration timeout when the LwM2M Registration ...
Read more >The Azure Cloud Native Architecture Mapbook - Dot.net
Stéphane Eyskens is an Azure solutions architect and a digital transformation advocate, helping organizations to get better results out of their cloud ...
Read more >Category Name - RSSing.com
Right-click your device and select Send D2C message to IoT Hub, then enter the message, results will be shown in OUTPUT > Azure...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Hi Sebastian. Yes, we are very well aware of the issues with Direct Methods. The root cause is that the AMQP link between the Edge Hub and the IoT Hub is being dropped silently. The Edge Hub relies on the C# SDK for this connectivity, and restarting the Edge Hub re-establishes this link, fixing the issue.
This behavior on AMQP has been fixed in 1.19.0 of the C# SDK, but this version has a couple of issues, preventing IoT Edge from using it. The PR mentioned above is a fix for one of these issues and will be merged. We will get this updated as soon as a fix comes, and are planning to get a release candidate out shortly.
This is the top priority bug to be fixed and the C# SDK team is working hard to address. We are holding the
1.0.7release of IoT Edge until we get this fixed and can verify it.There has been significant progress on the underlying issue and a long haul test plan is being run to verify. We are targeting a 1.0.7 release candidate by the end of the month but this is dependent on how this testing goes.