EdgeHub stops sending messages upstream and messages are not buffered locally.

After updating to IotEdge 1.0.7 our IotHub did not receive any messages anymore after a couple of days. The IotHubDevices Extension in VS Code displayed edgeAgent and edgeHub as connected but all custom modules as disconnected.

The edgeHub logs contained the following Message every two seconds (the frequency of messages send):

2019-05-13 03:32:03.923 +00:00 [WRN] - Non retryable exception occurred while sending message.
System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'IoT Client'.
   at Microsoft.Azure.Devices.Client.Transport.DefaultDelegatingHandler.ThrowIfDisposed()
   at Microsoft.Azure.Devices.Client.Transport.DefaultDelegatingHandler.OpenAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.<>c__DisplayClass32_0.<<OpenAsyncInternal>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.EnsureOpenedAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.<>c__DisplayClass14_0.<<SendEventAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.SendEventAsync(Message message, CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.InternalClient.SendEventAsync(Message message)
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.<>c__DisplayClass28_0.<<InvokeFunc>b__0>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 174
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient.InvokeFunc[T](Func`1 func, String operation, Boolean useForConnectivityCheck) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/ConnectivityAwareClient.cs:line 134
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudProxy.SendMessageAsync(IMessage inputMessage) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudProxy.cs:line 127
   at Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudProxy.SendMessageAsync(IMessage inputMessage) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.CloudProxy/CloudProxy.cs:line 134
   at Microsoft.Azure.Devices.Edge.Hub.Core.Routing.CloudEndpoint.CloudMessageProcessor.<>c__DisplayClass12_0.<<ProcessAsync>b__0>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Core/routing/CloudEndpoint.cs:line 146

In the edgeAgent logs I cannot see anything unusual.

I restarted edgeHub with

sudo iotedge restart edgeHub

but restarting failed with the following message repeating in the logs:

[2019-05-13 05:01:52 +00:00]: Starting Edge Hub
[05/13/2019 05:01:53.620 AM] Edge Hub Main()

Unhandled Exception: System.AggregateException: One or more errors occurred. (Error calling CreateServerCertificateAsync: Could not get server cert
	caused by: A error occurred in the key store.
	caused by: HSM failure
	caused by: HSM API returned an invalid null response) ---> Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadCommunicationException: Error calling CreateServerCertificateAsync: Could not get server cert
	caused by: A error occurred in the key store.
	caused by: HSM failure
	caused by: HSM API returned an invalid null response
   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2018_06_28.WorkloadClient.HandleException(Exception ex, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2018_06_28/WorkloadClient.cs:line 103
   at Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClientVersioned.Execute[T](Func`1 func, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/WorkloadClientVersioned.cs:line 56
   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2018_06_28.WorkloadClient.CreateServerCertificateAsync(String hostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2018_06_28/WorkloadClient.cs:line 32
   at Microsoft.Azure.Devices.Edge.Util.CertificateHelper.GetServerCertificatesFromEdgelet(Uri workloadUri, String workloadApiVersion, String workloadClientApiVersion, String moduleId, String moduleGenerationId, String edgeHubHostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/CertificateHelper.cs:line 249
   at Microsoft.Azure.Devices.Edge.Hub.Service.EdgeHubCertificates.LoadAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/EdgeHubCertificates.cs:line 53
   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.MainAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 49
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at System.Threading.Tasks.Task`1.get_Result()
   at Microsoft.Azure.Devices.Edge.Hub.Service.Program.Main() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 30
[2019-05-13 05:02:48 +00:00]: Starting Edge Hub
[05/13/2019 05:02:49.134 AM] Edge Hub Main()

With

sudo systemctl restart iotedge

all modules came up running again but all messages created since the last message was received by IotHub before the problem occured were lost !

specs raspberry pi iotedge 1.0.7 (f455ae2cd66db716c4bd8e7aaa7984ce092481c0) edgeAgent Version - 1.0.7.21905529 (f455ae2cd66db716c4bd8e7aaa7984ce092481c0) edgeHub Version - 1.0.7.21905529 (f455ae2cd66db716c4bd8e7aaa7984ce092481c0)

[varunpuranik]

Restarting the daemon should re-create the cert.

“docker ps” gives you when the container was “created”, so that might help. “docker inspect” has the timestamp.

The only time the edge runtime deletes a container is if it is updated. Is there a chance that happened?

The non-retryable exception is from the SDK, but the EdgeHub should retry the messages.

[jerfcero]

problem solved -> only run on VM console -> sudo systemctl restart iotedge

2021-08-13 15:32:51 Starting Edge Hub 2021-08-13 15:32:51.848 +00:00 Edge Hub Main() Unhandled exception. System.AggregateException: One or more errors occurred. (Error calling CreateServerCertificateAsync: Could not get server cert caused by: An error occurred creating the certificate caused by: HSM failure caused by: HSM API returned an invalid null response) —> Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadCommunicationException- Message:Error calling CreateServerCertificateAsync: Could not get server cert caused by: An error occurred creating the certificate caused by: HSM failure caused by: HSM API returned an invalid null response, StatusCode:500, at: at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.HandleException(Exception ex, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 109 at Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClientVersioned.Execute[T](Func1 func, String operation) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/WorkloadClientVersioned.cs:line 77 at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.CreateServerCertificateAsync(String hostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 35 at Microsoft.Azure.Devices.Edge.Util.CertificateHelper.GetServerCertificatesFromEdgelet(Uri workloadUri, String workloadApiVersion, String workloadClientApiVersion, String moduleId, String moduleGenerationId, String edgeHubHostname, DateTime expiration) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/CertificateHelper.cs:line 260 at Microsoft.Azure.Devices.Edge.Hub.Service.EdgeHubCertificates.LoadAsync(IConfigurationRoot configuration, ILogger logger) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/EdgeHubCertificates.cs:line 54 at Microsoft.Azure.Devices.Edge.Hub.Service.Program.MainAsync(IConfigurationRoot configuration) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 54 --- End of inner exception stack trace --- at System.Threading.Tasks.Task1.GetResultCore(Boolean waitCompletionNotification) at System.Threading.Tasks.Task`1.get_Result() at Microsoft.Azure.Devices.Edge.Hub.Service.Program.Main() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Service/Program.cs:line 33

NAME STATUS DESCRIPTION CONFIG edgeAgent running Up 10 minutes mcr.microsoft.com/azureiotedge-agent:1.0 edgeHub failed Failed (139) 4 minutes ago mcr.microsoft.com/azureiotedge-hub:1.0

NAME STATUS DESCRIPTION CONFIG edgeAgent running Up a minute mcr.microsoft.com/azureiotedge-agent:1.1 edgeHub failed Failed (139) 0 seconds ago mcr.microsoft.com/azureiotedge-hub:1.1

sudo systemctl restart iotedge

specs iotedge agents 1.1 iothub 1.1 linux vm 16.04 LTS