edgeAgent breaks with "System.ObjectDisposedException: Cannot access a disposed object."

Expected Behavior

Current Behavior

edgeAgent is not working and is constantly logging:

2019-06-08 11:23:08.930 +00:00 [WRN] - Reconcile failed because of the an exception
System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'IoT Client'.
   at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 116
2019-06-08 11:23:08.955 +00:00 [WRN] - Updating reported properties failed with error Cannot access a disposed object.
Object name: 'IoT Client'. type System.ObjectDisposedException

The edgeHub module and my own module is kept in the failed state by the edgeAgent.

Steps to Reproduce

I have not reproduced this.

Context (Environment)

Output of iotedge check --verbose

Click here

pi@edge5:~ $ sudo iotedge check  --verbose
Configuration checks
--------------------
√ config.yaml is well-formed
√ config.yaml has well-formed connection string
√ container engine is installed and functional
√ config.yaml has correct hostname
√ config.yaml has correct URIs for daemon mgmt endpoint
‼ latest security daemon
    Installed IoT Edge daemon has version 1.0.7 but version 1.0.7.1 is available.
    Please see https://aka.ms/iotedge-update-runtime for update instructions.
√ host time is close to real time
√ container time is close to host time
‼ DNS server
    Container engine is not configured with DNS server setting, which may impact connectivity to IoT Hub.
    Please see https://aka.ms/iotedge-prod-checklist-dns for best practices.
    You can ignore this warning if you are setting DNS server per module in the Edge deployment.
        caused by: Could not open container engine config file /etc/docker/daemon.json
        caused by: No such file or directory (os error 2)
√ production readiness: certificates
√ production readiness: certificates expiry
√ production readiness: container engine
‼ production readiness: logs policy
    Container engine is not configured to rotate module logs which may cause it run out of disk space.
    Please see https://aka.ms/iotedge-prod-checklist-logs for best practices.
    You can ignore this warning if you are setting log policy per module in the Edge deployment.
        caused by: Could not open container engine config file /etc/docker/daemon.json
        caused by: No such file or directory (os error 2)

Connectivity checks
-------------------
√ host can connect to and perform TLS handshake with IoT Hub AMQP port
√ host can connect to and perform TLS handshake with IoT Hub HTTPS port
√ host can connect to and perform TLS handshake with IoT Hub MQTT port
√ container on the default network can connect to IoT Hub AMQP port
√ container on the default network can connect to IoT Hub HTTPS port
√ container on the default network can connect to IoT Hub MQTT port
√ container on the IoT Edge module network can connect to IoT Hub AMQP port
√ container on the IoT Edge module network can connect to IoT Hub HTTPS port
√ container on the IoT Edge module network can connect to IoT Hub MQTT port
√ Edge Hub can bind to ports on host

One or more checks raised warnings.
pi@edge5:~ $ 

Device (Host) Operating System

Raspian 9.9

Architecture

arm32 (I think, uname -m returns armv7l)

Container Operating System

Linux containers

Runtime Versions

iotedged

iotedge 1.0.7 (f455ae2cd66db716c4bd8e7aaa7984ce092481c0)

Edge Agent

mcr.microsoft.com/azureiotedge-agent 1.0 5b352c655131 5 weeks ago 229MB

Edge Hub

mcr.microsoft.com/azureiotedge-hub 1.0 833a5058bd9b 5 weeks ago 244MB

Docker

pi@edge5:~ $ sudo docker version
Client:
 Version:           3.0.5
 API version:       1.40
 Go version:        go1.12.1
 Git commit:        ba9934d4
 Built:             Thu Apr 18 23:28:44 2019
 OS/Arch:           linux/arm
 Experimental:      false

Server:
 Engine:
  Version:          3.0.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.1
  Git commit:       dbe4a30
  Built:            Fri Apr 19 07:14:38 2019
  OS/Arch:          linux/arm
  Experimental:     false
 containerd:
  Version:          v1.2.5
  GitCommit:        bb71b10fd8f58240ca47fbb579b9d1028eea7c84
 runc:
  Version:          1.0.0-rc6+dev
  GitCommit:        2b18fe1d885ee5083ef9f0838fee39b62d653e30
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
pi@edge5:~ $ 

Logs

iotedged logs

pi@edge5:~ $ systemctl status -af iotedge
● iotedge.service - Azure IoT Edge daemon
   Loaded: loaded (/lib/systemd/system/iotedge.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2019-06-07 10:10:48 UTC; 1 weeks 0 days ago
     Docs: man:iotedged(8)
 Main PID: 855 (iotedged)
      CPU: 31min 44.565s
   CGroup: /system.slice/iotedge.service
           └─855 /usr/bin/iotedged -c /etc/iotedge/config.yaml

Jun 14 13:51:47 edge5 iotedged[855]: 2019-06-14T13:51:47Z [INFO] - [mgmt] - - - [2019-06-14 13:51:47.310588258 UTC] "GET /modules?api-version=2019-01-30 HTTP/1.1" 200 OK 172
Jun 14 13:51:48 edge5 iotedged[855]: 2019-06-14T13:51:48Z [INFO] - Checking edge runtime status
Jun 14 13:51:48 edge5 iotedged[855]: 2019-06-14T13:51:48Z [INFO] - Edge runtime is running.
Jun 14 13:51:52 edge5 iotedged[855]: 2019-06-14T13:51:52Z [INFO] - [mgmt] - - - [2019-06-14 13:51:52.379656607 UTC] "GET /modules?api-version=2019-01-30 HTTP/1.1" 200 OK 172
Jun 14 13:51:57 edge5 iotedged[855]: 2019-06-14T13:51:57Z [INFO] - [mgmt] - - - [2019-06-14 13:51:57.452065056 UTC] "GET /modules?api-version=2019-01-30 HTTP/1.1" 200 OK 172
Jun 14 13:52:02 edge5 iotedged[855]: 2019-06-14T13:52:02Z [INFO] - [mgmt] - - - [2019-06-14 13:52:02.531761747 UTC] "GET /modules?api-version=2019-01-30 HTTP/1.1" 200 OK 172
Jun 14 13:52:07 edge5 iotedged[855]: 2019-06-14T13:52:07Z [INFO] - [mgmt] - - - [2019-06-14 13:52:07.619273395 UTC] "GET /modules?api-version=2019-01-30 HTTP/1.1" 200 OK 172
Jun 14 13:52:12 edge5 iotedged[855]: 2019-06-14T13:52:12Z [INFO] - [mgmt] - - - [2019-06-14 13:52:12.691543377 UTC] "GET /modules?api-version=2019-01-30 HTTP/1.1" 200 OK 172
Jun 14 13:52:17 edge5 iotedged[855]: 2019-06-14T13:52:17Z [INFO] - [mgmt] - - - [2019-06-14 13:52:17.761779339 UTC] "GET /modules?api-version=2019-01-30 HTTP/1.1" 200 OK 172
Jun 14 13:52:22 edge5 iotedged[855]: 2019-06-14T13:52:22Z [INFO] - [mgmt] - - - [2019-06-14 13:52:22.845418073 UTC] "GET /modules?api-version=2019-01-30 HTTP/1.1" 200 OK 172
lines 1-19/19 (END)

edge-agent logs

agent.log

Problem starts around 2019-06-08 10:22, that is line 2440 in the attached log.

edge-hub logs

<Paste here>

Additional Information

[varunpuranik]

@vladislav-mitev - We have added logic to handle those ObjectDisposedExceptions in EdgeHub and EdgeAgent.

We update the IoTHub SDK in EdgeHub and EdgeAgent at regular intervals after proper testing. So it is difficult to say after how long, but we will try to take the updated SDK with the fix as soon as possible.

[arsing]

@varunpuranik can answer that question.