iotedged gives Internal server error: Module runtime error upon running with or without custom modules

I started off here however they asked for me to move my issue posting to this location. Thank you for your help.

Has information similar to what is going on with my raspberry pi 3 B+ system.

The command sudo systemctl status iotedge give the output:

● iotedge.service - Azure IoT Edge daemon
   Loaded: loaded (/lib/systemd/system/iotedge.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2018-08-06 15:36:16 UTC; 5h 14min ago
     Docs: man:iotedged(8)
 Main PID: 521 (iotedged)
      CPU: 32.742s
   CGroup: /system.slice/iotedge.service
           └─521 /usr/bin/iotedged -c /etc/iotedge/config.yaml

Aug 06 20:50:42 demo iotedged[521]: 2018-08-06T20:50:42Z [ERR!] - Internal server error: Module runtime error
Aug 06 20:50:42 demo iotedged[521]:         caused by: Container runtime error
Aug 06 20:50:42 demo iotedged[521]:         caused by: Serde error
Aug 06 20:50:42 demo iotedged[521]:         caused by: invalid type: sequence, expected a string at line 1 column 1137
Aug 06 20:50:42 demo iotedged[521]: 2018-08-06T20:50:42Z [INFO] - [mgmt] - - - [2018-08-06 20:50:42.083427884 UTC] "GET /modules?api-version=2018-06-28 HTTP/1.1" 500 Internal Server Error 176 "-" "-" pid(992)
Aug 06 20:50:47 demo iotedged[521]: 2018-08-06T20:50:47Z [ERR!] - Internal server error: Module runtime error
Aug 06 20:50:47 demo iotedged[521]:         caused by: Container runtime error
Aug 06 20:50:47 demo iotedged[521]:         caused by: Serde error
Aug 06 20:50:47 demo iotedged[521]:         caused by: invalid type: sequence, expected a string at line 1 column 1137
Aug 06 20:50:47 demo iotedged[521]: 2018-08-06T20:50:47Z [INFO] - [mgmt] - - - [2018-08-06 20:50:47.456599219 UTC] "GET /modules?api-version=2018-06-28 HTTP/1.1" 500 Internal Server Error 176 "-" "-" pid(992)

sudo iotedge logs edgeAgent has output that follows:

2018-08-06 20:58:35.463 +00:00 [WRN] - Reconcile failed because of the an exception
Microsoft.Azure.Devices.Edge.Agent.Edgelet.EdgeletCommunicationException- Message:Error calling List modules: Module runtime error
        caused by: Container runtime error
        caused by: Serde error
        caused by: invalid type: sequence, expected a string at line 1 column 1137, StatusCode:500, at:   at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in /opt/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 159
2018-08-06 20:58:57.889 +00:00 [WRN] - Reconcile failed because of the an exception
Microsoft.Azure.Devices.Edge.Agent.Edgelet.EdgeletCommunicationException- Message:Error calling List modules: Module runtime error
        caused by: Container runtime error
        caused by: Serde error
        caused by: invalid type: sequence, expected a string at line 1 column 1137, StatusCode:500, at:   at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in /opt/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 159
2018-08-06 20:59:20.923 +00:00 [WRN] - Reconcile failed because of the an exception
Microsoft.Azure.Devices.Edge.Agent.Edgelet.EdgeletCommunicationException- Message:Error calling List modules: Module runtime error
        caused by: Container runtime error
        caused by: Serde error
        caused by: invalid type: sequence, expected a string at line 1 column 1137, StatusCode:500, at:   at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in /opt/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 159
2018-08-06 20:59:41.850 +00:00 [WRN] - Reconcile failed because of the an exception
Microsoft.Azure.Devices.Edge.Agent.Edgelet.EdgeletCommunicationException- Message:Error calling List modules: Module runtime error
        caused by: Container runtime error
        caused by: Serde error
        caused by: invalid type: sequence, expected a string at line 1 column 1137, StatusCode:500, at:   at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in /opt/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 159

sudo iotedge list outputs:

An error in the management http client occurred.
        caused by: Client error

docker ps -a (installed from moby) gives:

CONTAINER ID        IMAGE                                                           COMMAND                   CREATED             STATUS              PORTS                                                                  NAMES
0123f76f1b98        demob:1.0.0                                                    "/usr/sbin/mosquitto…"    2 days ago          Up 5 hours          0.0.0.0:1883-1884->1883-1884/tcp                                       demo-b
a0bce36e8987        demom:1.0.0                                                    "npm start"                2 days ago          Up 5 hours          1884/tcp, 3000/tcp                                                     demo-m
10568d38ca85        mcr.microsoft.com/azureiotedge-hub:1.0                          "/bin/sh -c 'echo \"$…"   2 days ago          Up 5 hours          0.0.0.0:443->443/tcp, 0.0.0.0:5671->5671/tcp, 0.0.0.0:8883->8883/tcp   edgeHub
84b0d675e621        mcr.microsoft.com/azureiotedge-agent:1.0                        "/bin/sh -c 'echo \"$…"   2 days ago          Up 12 minutes                                                                              edgeAgent

Am I missing something I should be looking at? Is this behavior normal?

Thank you for your help in advance.

[dsajanice]

We have seen a similar issue before. It was fixed here: https://github.com/Azure/iotedge/commit/31468a1ec03d5b2d1077064563db4b853a961eab. The issue is observed when someone is exec’ed in to a container and iotedge is started. Is that the case with your setup?

[treedust]

@myagley is there a document of the projected release versions and what they will include? Such as bug fixes. As well will there documents of versions? Currently Azure Portal dictates a version to use; however, one can change the tagged image version and settings. But without knowing the tags and what the version includes or does not would result in unexpected behavior.