AmqpWs proxy authentication - log file reports credential usage; nonetheless tcpdump shows that authentication isn't working

I’m behind a corporate proxy so I configured the edge node as suggested (https://docs.microsoft.com/en-us/azure/iot-edge/how-to-configure-proxy-support). However I doubt that the the proxy_host in format user:password@proxy_host is used correctly everywhere - still it should work as https://docs.microsoft.com/en-us/azure/iot-edge/how-to-configure-proxy-support documents.

/etc/iotedge/config.yaml:

https_proxy: “http://user:password@proxy-server:8080/”
UpstreamProtocol: “AmqpWs”

Logging of the azure IOT Edge container:

2020-07-08 08:02:55.712 +00:00 Edge Agent Main()
<6> 2020-07-08 08:02:55.924 +00:00 [INF] - Initializing Edge Agent.
<6> 2020-07-08 08:02:56.111 +00:00 [INF] - Version - 1.0.9.3.32101608 (ad04e23b1865211913282b918263700deafdccfa)
<6> 2020-07-08 08:02:56.112 +00:00 [INF] -
        █████╗ ███████╗██╗   ██╗██████╗ ███████╗
       ██╔══██╗╚══███╔╝██║   ██║██╔══██╗██╔════╝
       ███████║  ███╔╝ ██║   ██║██████╔╝█████╗
       ██╔══██║ ███╔╝  ██║   ██║██╔══██╗██╔══╝
       ██║  ██║███████╗╚██████╔╝██║  ██║███████╗
       ╚═╝  ╚═╝╚══════╝ ╚═════╝ ╚═╝  ╚═╝╚══════╝

 ██╗ ██████╗ ████████╗    ███████╗██████╗  ██████╗ ███████╗
 ██║██╔═══██╗╚══██╔══╝    ██╔════╝██╔══██╗██╔════╝ ██╔════╝
 ██║██║   ██║   ██║       █████╗  ██║  ██║██║  ███╗█████╗
 ██║██║   ██║   ██║       ██╔══╝  ██║  ██║██║   ██║██╔══╝
 ██║╚██████╔╝   ██║       ███████╗██████╔╝╚██████╔╝███████╗
 ╚═╝ ╚═════╝    ╚═╝       ╚══════╝╚═════╝  ╚═════╝ ╚══════╝

<6> 2020-07-08 08:02:56.136 +00:00 [INF] - Detected proxy http://[user]:****@[proxy]:8080/
<6> 2020-07-08 08:02:56.183 +00:00 [INF] - Experimental features configuration: {"Enabled":false,"DisableCloudSubscriptions":false,"EnableUploadLogs":false,"EnableGetLogs":false,"EnableMetrics":false}
<6> 2020-07-08 08:02:56.353 +00:00 [INF] - Installing certificates [CN=Test Edge Owner CA:09/21/2020 08:44:29] to Root
<6> 2020-07-08 08:02:56.838 +00:00 [INF] - Created persistent store at /tmp/edgeAgent
<6> 2020-07-08 08:02:56.913 +00:00 [INF] - Started operation refresh twin config
<6> 2020-07-08 08:02:56.936 +00:00 [INF] - Edge agent attempting to connect to IoT Hub via Amqp_WebSocket_Only...
<6> 2020-07-08 08:02:57.003 +00:00 [INF] - Registering request handler RestartModule
<4> 2020-07-08 08:03:02.160 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup file (backup.json) instead
<6> 2020-07-08 08:03:02.167 +00:00 [INF] - Edge agent config backup file does not exist - backup.json
<4> 2020-07-08 08:03:12.187 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup file (backup.json) instead
<6> 2020-07-08 08:03:12.187 +00:00 [INF] - Edge agent config backup file does not exist - backup.json
<4> 2020-07-08 08:03:22.189 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup file (backup.json) instead
<6> 2020-07-08 08:03:22.189 +00:00 [INF] - Edge agent config backup file does not exist - backup.json
<4> 2020-07-08 08:03:32.190 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup file (backup.json) instead
<6> 2020-07-08 08:03:32.190 +00:00 [INF] - Edge agent config backup file does not exist - backup.json
<6> 2020-07-08 08:03:33.233 +00:00 [INF] - Edge agent attempting to connect to IoT Hub via Amqp_WebSocket_Only...
<4> 2020-07-08 08:03:42.191 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup file (backup.json) instead
<6> 2020-07-08 08:03:42.191 +00:00 [INF] - Edge agent config backup file does not exist - backup.json
<4> 2020-07-08 08:03:52.192 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup file (backup.json) instead
<6> 2020-07-08 08:03:52.192 +00:00 [INF] - Edge agent config backup file does not exist - backup.json
<4> 2020-07-08 08:04:02.194 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup file (backup.json) instead
<6> 2020-07-08 08:04:02.194 +00:00 [INF] - Edge agent config backup file does not exist - backup.json
<4> 2020-07-08 08:04:12.196 +00:00 [WRN] - Empty edge agent config was received. Attempting to read config from backup file (backup.json) instead
<6> 2020-07-08 08:04:12.196 +00:00 [INF] - Edge agent config backup file does not exist - backup.json

If I take a look at the tcpdump I see that the proxy is answering with a “HTTP/1.1 407 Proxy Authentication Required” to the connect request of the edgeAgent container - but the connect from the edgeAgent doesn’t provide an authentication header (also it should).

My guess is that the edge correctly detects the proxy credentials (proofed by the logging line “Detected proxy http://[user]😗***@[proxy]:8080/”) but for whatever reason doesn’t use the provided credentials to connect via AmqpWs to the IotHub.

Expected Behavior

EdgeAgent should be able to connect to the cloud via amqp-ws trough the proxy.

Current Behavior

Credentials didn’t seem to make it to the connect telegram to the proxy.

Steps to Reproduce

Setup a proxy that enforces basic authentication; or just configure username/password and dump the CONNECT telegram of the AmqpWs connection.

Context (Environment)

Device Information

• Host OS: RHEL7
• Architecture: x86_64
• Container OS: Linux containers

Runtime Versions

• iotedged: iotedge 1.0.9.3
• Edge Agent: mcr.microsoft.com/azureiotedge-agent:1.0.9.3
• Docker/Moby:

 Version:           3.0.13+azure
 API version:       1.40
 Go version:        go1.13.11
 Git commit:        dd360c7c0de8d9132a3965db6a59d3ae74f43ba7
 Built:             Thu May 28 20:41:25 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          3.0.13+azure
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.11
  Git commit:       77e06fda0c
  Built:            Mon Jun  1 20:22:00 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Logs

See information above.

[ghost]

My apologies, I should have added some front-matter explaining my comment. I agree wholeheartedly that this is a problem; my intention was to provide some additional background for other developers on the project.

I will open an issue in the SDK repository (right after I finish my coffee) and link it here for tracking.

Edit to add: One thing that I should mention is that this area is somewhat outside of my purview. I probably cannot be of too much further assistance for this reason. I can still keep an eye on this issue since I have interacted with it, but I am afraid that my knowledge in this area is limited.

[github-actions[bot]]

This issue is being marked as stale because it has been open for 30 days with no activity.