tutorial-develop-for-linux - Container start fails with error 500

Expected Behavior

Follow the tutorial at https://learn.microsoft.com/en-us/azure/iot-edge/tutorial-develop-for-linux?view=iotedge-1.4. Expect the tutorial to work as expected, and the SampleModule to start on the VM.

Current Behavior

Module is registered on VM, but cannot start. When you check in the Azure Edge Agent logs on the backend, you get the following error: System.AggregateException: One or more errors occurred. (Error calling prepare update for module SampleModule: registry operation error: pull image “XXXcontainerrepo.azurecr.io/samplemodule:0.0.1-amd64”) —> Microsoft.Azure.Devices.Edge.Agent.Edgelet.EdgeletCommunicationException- Message:Error calling prepare update for module SampleModule: registry operation error: pull image “XXXcontainerrepo.azurecr.io/samplemodule:0.0.1-amd64”, StatusCode:500, at: at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2021_12_07.ModuleManagementHttpClient.HandleException(Exception exception, String operation) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2021_12_07/ModuleManagementHttpClient.cs:line 235 at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Versioning.ModuleManagementHttpClientVersioned.Execute[T](Func1 func, String operation) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/versioning/ModuleManagementHttpClientVersioned.cs:line 153 at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2021_12_07.ModuleManagementHttpClient.PrepareUpdateAsync(ModuleSpec moduleSpec) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2021_12_07/ModuleManagementHttpClient.cs:line 214 at Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient.<>c__DisplayClass26_0.<<Throttle>b__0>d.MoveNext() in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/ModuleManagementHttpClient.cs:line 139 --- End of stack trace from previous location where exception was thrown --- at Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient.Throttle[T](Func1 identityOperation) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/ModuleManagementHttpClient.cs:line 153 at Microsoft.Azure.Devices.Edge.Agent.Core.Commands.GroupCommand.ExecuteAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/commands/GroupCommand.cs:line 35 at Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory.LoggingCommand.ExecuteAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/LoggingCommandFactory.cs:line 60 at Microsoft.Azure.Devices.Edge.Agent.Core.Commands.GroupCommand.ExecuteAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/commands/GroupCommand.cs:line 35 at Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory.LoggingCommand.ExecuteAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/LoggingCommandFactory.cs:line 60 at Microsoft.Azure.Devices.Edge.Agent.Core.PlanRunners.OrderedRetryPlanRunner.ExecuteAsync(Int64 deploymentId, Plan plan, CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/planrunners/OrdererdRetryPlanRunner.cs:line 87

This is very weird as I manually authorized inside the VM using docker login, and was able to pull the image successfully even.

Steps to Reproduce

Provide a detailed set of steps to reproduce the bug.

1. Follow tutorial at https://learn.microsoft.com/en-us/azure/iot-edge/tutorial-develop-for-linux?view=iotedge-1.4.

Context (Environment)

HOST: Ubuntu 20.04LTS Edge IoT Device: VM running Ubuntu 20.04LTS. Setup was done with version 1.4, but the deployment of the containers downgraded it to 1.2.

Output of iotedge check

Showed no error

Click here

<Paste here between the triple backticks>

Device Information

• Host OS [e.g. Ubuntu 18.04, Windows Server IoT 2019]:
• Architecture [e.g. amd64, arm32, arm64]:
• Container OS [e.g. Linux containers, Windows containers]:

Runtime Versions

• aziot-edged [run iotedge version]:
• Edge Agent [image tag (e.g. 1.0.0)]:
• Edge Hub [image tag (e.g. 1.0.0)]:
• Docker/Moby [run docker version]:

Note: when using Windows containers on Windows, run docker -H npipe:////./pipe/iotedge_moby_engine version instead

Logs

aziot-edged logs

<Paste here between the triple backticks>

edge-agent logs

<Paste here between the triple backticks>

edge-hub logs

<Paste here between the triple backticks>

Additional Information

Please provide any additional information that may be helpful in understanding the issue.

[TAMHAN]

Hello, sorry for being slow. I am still in West Germany, away from my deployment which lives on the machine in Budapest.

I promise, when I come back, I look at it immediately.

On November 23, 2022 12:22:28 AM GMT+01:00, Robert T Jang @.***> wrote:

@TAMHAN ping again 😃

– Reply to this email directly or view it on GitHub: https://github.com/Azure/iotedge/issues/6779#issuecomment-1324360384 You are receiving this because you were mentioned.

Message ID: @.***>

With best regards Tam HANNA

– Enjoy electronics, 3D printing and cigars? Join more than 21000 followers on my Instagram at http://www.instagram.com/tam.hanna

[kuan51]

I seem to have run into the same issue as well. I double checked the registry credentials in my deployment json and the credentials are correct. My heartbeat module is just the generated C# IoT module template with no modifications.

iotedge check output

  Configuration checks (aziot-identity-service)
  ---------------------------------------------
  √ keyd configuration is well-formed - OK
  √ certd configuration is well-formed - OK
  √ tpmd configuration is well-formed - OK
  √ identityd configuration is well-formed - OK
  √ daemon configurations up-to-date with config.toml - OK
  √ identityd config toml file specifies a valid hostname - OK
  √ aziot-identity-service package is up-to-date - OK
  √ host time is close to reference time - OK
  √ preloaded certificates are valid - OK
  √ keyd is running - OK
  √ certd is running - OK
  √ identityd is running - OK
  √ read all preloaded certificates from the Certificates Service - OK
  √ read all preloaded key pairs from the Keys Service - OK
  √ check all EST server URLs utilize HTTPS - OK
  √ ensure all preloaded certificates match preloaded private keys with the same ID - OK
  
  Connectivity checks (aziot-identity-service)
  --------------------------------------------
  √ host can connect to and perform TLS handshake with iothub AMQP port - OK
  √ host can connect to and perform TLS handshake with iothub HTTPS / WebSockets port - OK
  √ host can connect to and perform TLS handshake with iothub MQTT port - OK
  
  Configuration checks
  --------------------
  √ aziot-edged configuration is well-formed - OK
  √ configuration up-to-date with config.toml - OK
  √ container engine is installed and functional - OK
  √ configuration has correct URIs for daemon mgmt endpoint - OK
  √ aziot-edge package is up-to-date - OK
  √ container time is close to host time - OK
  √ DNS server - OK
  √ production readiness: logs policy - OK
  ‼ production readiness: Edge Agent's storage directory is persisted on the host filesystem - Warning
      The edgeAgent module is not configured to persist its /tmp/edgeAgent directory on the host filesystem.
      Data might be lost if the module is deleted or updated.
      Please see https://aka.ms/iotedge-storage-host for best practices.
  ‼ production readiness: Edge Hub's storage directory is persisted on the host filesystem - Warning
      The edgeHub module is not configured to persist its /tmp/edgeHub directory on the host filesystem.
      Data might be lost if the module is deleted or updated.
      Please see https://aka.ms/iotedge-storage-host for best practices.
  √ Agent image is valid and can be pulled from upstream - OK
  √ proxy settings are consistent in aziot-edged, aziot-identityd, moby daemon and config.toml - OK
  
  Connectivity checks
  -------------------
  √ container on the default network can connect to upstream AMQP port - OK
  √ container on the default network can connect to upstream HTTPS / WebSockets port - OK
  √ container on the IoT Edge module network can connect to upstream AMQP port - OK
  √ container on the IoT Edge module network can connect to upstream HTTPS / WebSockets port - OK
  33 check(s) succeeded.
  2 check(s) raised warnings. Re-run with --verbose for more details.
  2 check(s) were skipped due to errors from other checks. Re-run with --verbose for more details.

edgeAgent Logs Excerpt

<6> 2023-04-21 22:38:13.504 +00:00 [INF] - Executing command: "Command Group: (\n  [Command Group: (\n  [Prepare module heartbeatmodule]\n  [Create module heartbeatmodule]\n)]\n  [Start module heartbeatmodule]\n)"
  <6> 2023-04-21 22:38:13.504 +00:00 [INF] - Executing command: "Command Group: (\n  [Prepare module heartbeatmodule]\n  [Create module heartbeatmodule]\n)"
  <3> 2023-04-21 22:38:14.747 +00:00 [ERR] - Executing command for operation ["create"] failed.
  Microsoft.Azure.Devices.Edge.Agent.Edgelet.EdgeletCommunicationException- Message:Error calling prepare update for module heartbeatmodule: registry operation error: pull image "xenfidev.azurecr.io/heartbeatmodule:latest", StatusCode:404, at:   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2022_08_03.ModuleManagementHttpClient.HandleException(Exception exception, String operation) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2022_08_03/ModuleManagementHttpClient.cs:line 232
     at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Versioning.ModuleManagementHttpClientVersioned.Execute[T](Func`1 func, String operation) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/versioning/ModuleManagementHttpClientVersioned.cs:line 168
     at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2022_08_03.ModuleManagementHttpClient.PrepareUpdateAsync(ModuleSpec moduleSpec) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2022_08_03/ModuleManagementHttpClient.cs:line 214
     at Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient.<>c__DisplayClass26_0.<<Throttle>b__0>d.MoveNext() in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/ModuleManagementHttpClient.cs:line 143
  --- End of stack trace from previous location ---
     at Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient.Throttle[T](Func`1 identityOperation) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/ModuleManagementHttpClient.cs:line 158
     at Microsoft.Azure.Devices.Edge.Agent.Core.Commands.GroupCommand.ExecuteAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/commands/GroupCommand.cs:line 35
     at Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory.LoggingCommand.ExecuteAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/LoggingCommandFactory.cs:line 61
  <3> 2023-04-21 22:38:14.748 +00:00 [ERR] - Executing command for operation ["Command Group: (\n  [Command Group: (\n  [Prepare module heartbeatmodule]\n  [Create module heartbeatmodule]\n)]\n  [Start module heartbeatmodule]\n)"] failed.
  Microsoft.Azure.Devices.Edge.Agent.Edgelet.EdgeletCommunicationException- Message:Error calling prepare update for module heartbeatmodule: registry operation error: pull image "xenfidev.azurecr.io/heartbeatmodule:latest", StatusCode:404, at:   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2022_08_03.ModuleManagementHttpClient.HandleException(Exception exception, String operation) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2022_08_03/ModuleManagementHttpClient.cs:line 232
     at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Versioning.ModuleManagementHttpClientVersioned.Execute[T](Func`1 func, String operation) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/versioning/ModuleManagementHttpClientVersioned.cs:line 168
     at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2022_08_03.ModuleManagementHttpClient.PrepareUpdateAsync(ModuleSpec moduleSpec) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2022_08_03/ModuleManagementHttpClient.cs:line 214
     at Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient.<>c__DisplayClass26_0.<<Throttle>b__0>d.MoveNext() in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/ModuleManagementHttpClient.cs:line 143
  --- End of stack trace from previous location ---
     at Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient.Throttle[T](Func`1 identityOperation) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/ModuleManagementHttpClient.cs:line 158
     at Microsoft.Azure.Devices.Edge.Agent.Core.Commands.GroupCommand.ExecuteAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/commands/GroupCommand.cs:line 35
     at Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory.LoggingCommand.ExecuteAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/LoggingCommandFactory.cs:line 61
     at Microsoft.Azure.Devices.Edge.Agent.Core.Commands.GroupCommand.ExecuteAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/commands/GroupCommand.cs:line 35
     at Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory.LoggingCommand.ExecuteAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/LoggingCommandFactory.cs:line 61
  <3> 2023-04-21 22:38:14.749 +00:00 [ERR] - Step failed in deployment 9, continuing execution. Failure when running command Command Group: (
    [Command Group: (
    [Prepare module heartbeatmodule]
    [Create module heartbeatmodule]
  )]
    [Start module heartbeatmodule]
  ). Will retry in -03s.
  <6> 2023-04-21 22:38:14.749 +00:00 [INF] - Plan execution ended for deployment 9
  <3> 2023-04-21 22:38:14.749 +00:00 [ERR] - Edge agent plan execution failed.
  System.AggregateException: One or more errors occurred. (Error calling prepare update for module heartbeatmodule: registry operation error: pull image "xenfidev.azurecr.io/heartbeatmodule:latest")
   ---> Microsoft.Azure.Devices.Edge.Agent.Edgelet.EdgeletCommunicationException- Message:Error calling prepare update for module heartbeatmodule: registry operation error: pull image "xenfidev.azurecr.io/heartbeatmodule:latest", StatusCode:404, at:   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2022_08_03.ModuleManagementHttpClient.HandleException(Exception exception, String operation) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2022_08_03/ModuleManagementHttpClient.cs:line 232
     at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Versioning.ModuleManagementHttpClientVersioned.Execute[T](Func`1 func, String operation) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/versioning/ModuleManagementHttpClientVersioned.cs:line 168
     at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2022_08_03.ModuleManagementHttpClient.PrepareUpdateAsync(ModuleSpec moduleSpec) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2022_08_03/ModuleManagementHttpClient.cs:line 214
     at Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient.<>c__DisplayClass26_0.<<Throttle>b__0>d.MoveNext() in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/ModuleManagementHttpClient.cs:line 143
  --- End of stack trace from previous location ---

EDIT: I was able to resolve my issue. I was using version tags with my container images, but I never made a “latest” tag. Thus my container manifest never had an entry it could resolve for latest. I would include the explicit version tag in my IoT Edge deployment but it would just get removed and default to looking for the “latest” tag which didnt exist, thus the 404 error when trying to pull the container. Making sure I tagged the most recent container with both its explicit version and “latest” fixed the issue and I was able to push modules to my IoT edge devices.