Unable to recieve with IoTHubModuleClient

I’ve deployed a module with code that is essentially copy and pasted from the advanced-edge-scenarios sample:

import asyncio
from six.moves import input
import threading
from azure.iot.device.aio import IoTHubModuleClient

import logging

async def main():
    # Basic configuration: configure the root logger, including 'azure.storage'
    logging.basicConfig(
        format='%(asctime)s %(name)-20s %(levelname)-5s %(message)s',
        level=logging.DEBUG
    )
    # The client object is used to interact with your Azure IoT hub.
    module_client = IoTHubModuleClient.create_from_edge_environment()
    # connect the client.
    await module_client.connect()

    # define behavior for receiving an input message on input1
    async def input1_listener(module_client):
        while True:
            input_message = await module_client.receive_message_on_input(
                "input1"
            )  # blocking call
            print("the data in the message received on input1 was ")
            print(input_message.data)
            print("custom properties are")
            print(input_message.custom_properties)

    # define behavior for halting the application
    def stdin_listener():
        while True:
            selection = input("Press Q to quit\n")
            if selection == "Q" or selection == "q":
                print("Quitting...")
                break

    # Schedule task for listeners
    listeners = asyncio.gather(
        input1_listener(module_client)
    )
    # Run the stdin listener in the event loop
    loop = asyncio.get_running_loop()
    user_finished = loop.run_in_executor(None, stdin_listener)
    # Wait for user to indicate they are done listening for messages
    await user_finished
    # Cancel listening
    listeners.cancel()
    # Finally, disconnect
    await module_client.disconnect()


if __name__ == "__main__":
    # asyncio.run(main())
    # If using Python 3.6 or below, use the following code instead of asyncio.run(main()):
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())
    loop.close()

The container runs on Ubuntu 18.04 and uses Python3.6. I’m also using the latest version of the preview package (2.0.0rc11). Here is the Dockerfile:

FROM ubuntu:18.04

WORKDIR /app

RUN apt-get update && \
    apt-get install -y --no-install-recommends python3-setuptools python3-pip python3 && \
    rm -rf /var/lib/apt/lists/* 

RUN pip3 install --upgrade pip
COPY requirements.txt ./
RUN pip3 install --no-cache-dir -r requirements.txt && pip3 install --upgrade pip

COPY . .

RUN useradd -ms /bin/bash moduleuser
USER moduleuser

CMD [ "python3", "-u", "./main.py" ]

All seemingly straightforward. However, the module fails on the await module_client.connect() call. Here are the corresponding logs:

2019-10-07 17:57:46,410 azure.iot.device.iothub.auth.iotedge_authentication_provider INFO  Using IoTEdge authentication for {<iot-hub-url>, Staging_Gateway, avroProducer}
2019-10-07 17:57:46,413 urllib3.connectionpool DEBUG http://localhost:None "GET /trust-bundle?api-version=2019-01-30 HTTP/1.1" 200 2034
2019-10-07 17:57:46,414 azure.iot.device.common.pipeline.pipeline_thread DEBUG Starting run_op in pipeline thread
2019-10-07 17:57:46,414 azure.iot.device.common.pipeline.pipeline_thread DEBUG Creating pipeline executor
2019-10-07 17:57:46,415 azure.iot.device.common.pipeline.pipeline_stages_base DEBUG PipelineRootStage(SetAuthProviderOperation): running
2019-10-07 17:57:46,415 azure.iot.device.common.pipeline.operation_flow DEBUG PipelineRootStage(SetAuthProviderOperation): passing to next stage.
2019-10-07 17:57:46,415 azure.iot.device.common.pipeline.pipeline_stages_base DEBUG UseAuthProviderStage(SetAuthProviderOperation): running
2019-10-07 17:57:46,415 azure.iot.device.iothub.auth.base_renewable_token_authentication_provider INFO  Generating new SAS token for (Staging_Gateway,avroProducer) that expires 3600 seconds in the future
2019-10-07 17:57:46,431 urllib3.connectionpool DEBUG http://localhost:None "POST /modules/avroProducer/genid/637060084433818397/sign?api-version=2019-01-30 HTTP/1.1" 200 57
2019-10-07 17:57:46,432 azure.iot.device.iothub.auth.base_renewable_token_authentication_provider DEBUG Scheduling token update for (Staging_Gateway,avroProducer) for 3480 seconds in the future
2019-10-07 17:57:46,432 azure.iot.device.iothub.auth.base_renewable_token_authentication_provider DEBUG sending token update notification for (Staging_Gateway, avroProducer)
2019-10-07 17:57:46,432 azure.iot.device.common.pipeline.pipeline_thread DEBUG Already in pipeline thread for on_sas_token_updated
2019-10-07 17:57:46,432 azure.iot.device.iothub.pipeline.pipeline_stages_iothub INFO  UseAuthProviderStage: New sas token received.  Passing down UpdateSasTokenOperation.
2019-10-07 17:57:46,432 azure.iot.device.common.pipeline.operation_flow DEBUG UseAuthProviderStage(UpdateSasTokenOperation): passing to next stage.
2019-10-07 17:57:46,433 azure.iot.device.common.pipeline.pipeline_stages_base DEBUG HandleTwinOperationsStage(UpdateSasTokenOperation): running
2019-10-07 17:57:46,433 azure.iot.device.common.pipeline.operation_flow DEBUG HandleTwinOperationsStage(UpdateSasTokenOperation): passing to next stage.
2019-10-07 17:57:46,433 azure.iot.device.common.pipeline.pipeline_stages_base DEBUG CoordinateRequestAndResponseStage(UpdateSasTokenOperation): running
2019-10-07 17:57:46,433 azure.iot.device.common.pipeline.operation_flow DEBUG CoordinateRequestAndResponseStage(UpdateSasTokenOperation): passing to next stage.
2019-10-07 17:57:46,433 azure.iot.device.common.pipeline.pipeline_stages_base DEBUG IoTHubMQTTConverterStage(UpdateSasTokenOperation): running
2019-10-07 17:57:46,433 azure.iot.device.common.pipeline.operation_flow DEBUG IoTHubMQTTConverterStage(UpdateSasTokenOperation): passing to next stage.
2019-10-07 17:57:46,433 azure.iot.device.common.pipeline.pipeline_stages_base DEBUG EnsureConnectionStage(UpdateSasTokenOperation): running
2019-10-07 17:57:46,433 azure.iot.device.common.pipeline.operation_flow DEBUG EnsureConnectionStage(UpdateSasTokenOperation): passing to next stage.
2019-10-07 17:57:46,433 azure.iot.device.common.pipeline.pipeline_stages_base DEBUG SerializeConnectOpsStage(UpdateSasTokenOperation): running
2019-10-07 17:57:46,433 azure.iot.device.common.pipeline.operation_flow DEBUG SerializeConnectOpsStage(UpdateSasTokenOperation): passing to next stage.
2019-10-07 17:57:46,433 azure.iot.device.common.pipeline.pipeline_stages_base DEBUG MQTTTransportStage(UpdateSasTokenOperation): running
2019-10-07 17:57:46,433 azure.iot.device.common.pipeline.pipeline_stages_mqtt DEBUG MQTTTransportStage(UpdateSasTokenOperation): saving sas token and completing
2019-10-07 17:57:46,433 azure.iot.device.common.pipeline.operation_flow DEBUG MQTTTransportStage(UpdateSasTokenOperation): completing without error
2019-10-07 17:57:46,433 azure.iot.device.iothub.pipeline.pipeline_stages_iothub DEBUG UseAuthProviderStage(UpdateSasTokenOperation): token update operation is complete
2019-10-07 17:57:46,433 azure.iot.device.common.pipeline.operation_flow DEBUG UseAuthProviderStage(SetAuthProviderOperation): continuing with SetIoTHubConnectionArgsOperation op
2019-10-07 17:57:46,433 azure.iot.device.common.pipeline.operation_flow DEBUG UseAuthProviderStage(SetIoTHubConnectionArgsOperation): passing to next stage.
2019-10-07 17:57:46,433 azure.iot.device.common.pipeline.pipeline_stages_base DEBUG HandleTwinOperationsStage(SetIoTHubConnectionArgsOperation): running
2019-10-07 17:57:46,433 azure.iot.device.common.pipeline.operation_flow DEBUG HandleTwinOperationsStage(SetIoTHubConnectionArgsOperation): passing to next stage.
2019-10-07 17:57:46,433 azure.iot.device.common.pipeline.pipeline_stages_base DEBUG CoordinateRequestAndResponseStage(SetIoTHubConnectionArgsOperation): running
2019-10-07 17:57:46,433 azure.iot.device.common.pipeline.operation_flow DEBUG CoordinateRequestAndResponseStage(SetIoTHubConnectionArgsOperation): passing to next stage.
2019-10-07 17:57:46,433 azure.iot.device.common.pipeline.pipeline_stages_base DEBUG IoTHubMQTTConverterStage(SetIoTHubConnectionArgsOperation): running
2019-10-07 17:57:46,433 azure.iot.device.common.pipeline.operation_flow DEBUG IoTHubMQTTConverterStage(SetIoTHubConnectionArgsOperation): continuing with SetMQTTConnectionArgsOperation op
2019-10-07 17:57:46,434 azure.iot.device.common.pipeline.operation_flow DEBUG IoTHubMQTTConverterStage(SetMQTTConnectionArgsOperation): passing to next stage.
2019-10-07 17:57:46,434 azure.iot.device.common.pipeline.pipeline_stages_base DEBUG EnsureConnectionStage(SetMQTTConnectionArgsOperation): running
2019-10-07 17:57:46,434 azure.iot.device.common.pipeline.operation_flow DEBUG EnsureConnectionStage(SetMQTTConnectionArgsOperation): passing to next stage.
2019-10-07 17:57:46,434 azure.iot.device.common.pipeline.pipeline_stages_base DEBUG SerializeConnectOpsStage(SetMQTTConnectionArgsOperation): running
2019-10-07 17:57:46,434 azure.iot.device.common.pipeline.operation_flow DEBUG SerializeConnectOpsStage(SetMQTTConnectionArgsOperation): passing to next stage.
2019-10-07 17:57:46,434 azure.iot.device.common.pipeline.pipeline_stages_base DEBUG MQTTTransportStage(SetMQTTConnectionArgsOperation): running
2019-10-07 17:57:46,434 azure.iot.device.common.pipeline.pipeline_stages_mqtt DEBUG MQTTTransportStage(SetMQTTConnectionArgsOperation): got connection args
2019-10-07 17:57:46,434 azure.iot.device.common.mqtt_transport INFO  creating mqtt client
2019-10-07 17:57:46,434 azure.iot.device.common.mqtt_transport DEBUG creating a SSL context
2019-10-07 17:57:46,435 azure.iot.device.common.mqtt_transport DEBUG Created MQTT protocol client, assigned callbacks
2019-10-07 17:57:46,435 azure.iot.device.common.pipeline.operation_flow DEBUG MQTTTransportStage(SetMQTTConnectionArgsOperation): completing without error
2019-10-07 17:57:46,435 azure.iot.device.common.pipeline.operation_flow DEBUG IoTHubMQTTConverterStage(SetIoTHubConnectionArgsOperation): completing with result from SetMQTTConnectionArgsOperation
2019-10-07 17:57:46,435 azure.iot.device.common.pipeline.operation_flow DEBUG IoTHubMQTTConverterStage(SetIoTHubConnectionArgsOperation): completing without error
2019-10-07 17:57:46,435 azure.iot.device.common.pipeline.operation_flow DEBUG UseAuthProviderStage(SetAuthProviderOperation): completing with result from SetIoTHubConnectionArgsOperation
2019-10-07 17:57:46,435 azure.iot.device.common.pipeline.operation_flow DEBUG UseAuthProviderStage(SetAuthProviderOperation): completing without error
2019-10-07 17:57:46,435 azure.iot.device.common.pipeline.pipeline_thread DEBUG Starting <azure.iot.device.common.evented_callback.EventedCallback object at 0x7f81b2d545c0> in callback thread
2019-10-07 17:57:46,435 azure.iot.device.common.pipeline.pipeline_thread DEBUG Creating callback executor
2019-10-07 17:57:46,435 azure.iot.device.common.evented_callback DEBUG Callback completed with result None
2019-10-07 17:57:46,436 azure.iot.device.iothub.aio.async_clients INFO  Connecting to Hub...
2019-10-07 17:57:46,436 azure.iot.device.iothub.pipeline.iothub_pipeline DEBUG Starting ConnectOperation on the pipeline
2019-10-07 17:57:46,436 azure.iot.device.common.pipeline.pipeline_thread DEBUG Starting run_op in pipeline thread
2019-10-07 17:57:46,436 azure.iot.device.common.pipeline.pipeline_stages_base DEBUG PipelineRootStage(ConnectOperation): running
2019-10-07 17:57:46,436 azure.iot.device.common.pipeline.operation_flow DEBUG PipelineRootStage(ConnectOperation): passing to next stage.
2019-10-07 17:57:46,436 azure.iot.device.common.pipeline.pipeline_stages_base DEBUG UseAuthProviderStage(ConnectOperation): running
2019-10-07 17:57:46,436 azure.iot.device.common.pipeline.operation_flow DEBUG UseAuthProviderStage(ConnectOperation): passing to next stage.
2019-10-07 17:57:46,436 azure.iot.device.common.pipeline.pipeline_stages_base DEBUG HandleTwinOperationsStage(ConnectOperation): running
2019-10-07 17:57:46,436 azure.iot.device.common.pipeline.operation_flow DEBUG HandleTwinOperationsStage(ConnectOperation): passing to next stage.
2019-10-07 17:57:46,437 azure.iot.device.common.pipeline.pipeline_stages_base DEBUG CoordinateRequestAndResponseStage(ConnectOperation): running
2019-10-07 17:57:46,437 azure.iot.device.common.pipeline.operation_flow DEBUG CoordinateRequestAndResponseStage(ConnectOperation): passing to next stage.
2019-10-07 17:57:46,437 azure.iot.device.common.pipeline.pipeline_stages_base DEBUG IoTHubMQTTConverterStage(ConnectOperation): running
2019-10-07 17:57:46,437 azure.iot.device.common.pipeline.operation_flow DEBUG IoTHubMQTTConverterStage(ConnectOperation): passing to next stage.
2019-10-07 17:57:46,437 azure.iot.device.common.pipeline.pipeline_stages_base DEBUG EnsureConnectionStage(ConnectOperation): running
2019-10-07 17:57:46,437 azure.iot.device.common.pipeline.operation_flow DEBUG EnsureConnectionStage(ConnectOperation): passing to next stage.
2019-10-07 17:57:46,437 azure.iot.device.common.pipeline.pipeline_stages_base DEBUG SerializeConnectOpsStage(ConnectOperation): running
2019-10-07 17:57:46,437 azure.iot.device.common.pipeline.pipeline_stages_base DEBUG SerializeConnectOpsStage(ConnectOperation): blocking
2019-10-07 17:57:46,437 azure.iot.device.common.pipeline.operation_flow DEBUG SerializeConnectOpsStage(ConnectOperation): passing to next stage.
2019-10-07 17:57:46,437 azure.iot.device.common.pipeline.pipeline_stages_base DEBUG MQTTTransportStage(ConnectOperation): running
2019-10-07 17:57:46,437 azure.iot.device.common.pipeline.pipeline_stages_mqtt INFO  MQTTTransportStage(ConnectOperation): connecting
2019-10-07 17:57:46,437 azure.iot.device.common.mqtt_transport INFO  connecting to mqtt broker
2019-10-07 17:57:46,437 azure.iot.device.common.pipeline.pipeline_stages_mqtt ERROR transport.connect raised error
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/azure/iot/device/common/mqtt_transport.py", line 252, in connect
    rc = self._mqtt_client.connect(host=self._hostname, port=8883)
  File "/usr/local/lib/python3.6/dist-packages/paho/mqtt/client.py", line 839, in connect
    return self.reconnect()
  File "/usr/local/lib/python3.6/dist-packages/paho/mqtt/client.py", line 962, in reconnect
    sock = socket.create_connection((self._host, self._port), source_address=(self._bind_address, 0))
  File "/usr/lib/python3.6/socket.py", line 704, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  File "/usr/lib/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/azure/iot/device/common/pipeline/pipeline_stages_mqtt.py", line 100, in _execute_op
    self.transport.connect(password=self.sas_token)
  File "/usr/local/lib/python3.6/dist-packages/azure/iot/device/common/mqtt_transport.py", line 255, in connect
    message="Unexpected Paho failure during connect", cause=e
azure.iot.device.common.transport_exceptions.ProtocolClientError: Unexpected Paho failure during connect
2019-10-07 17:57:46,438 azure.iot.device.common.pipeline.operation_flow ERROR MQTTTransportStage(ConnectOperation): completing with error Unexpected Paho failure during connect
2019-10-07 17:57:46,438 azure.iot.device.common.pipeline.pipeline_stages_base ERROR SerializeConnectOpsStage(ConnectOperation): op failed.  Unblocking queue with error: Unexpected Paho failure during connect
2019-10-07 17:57:46,439 azure.iot.device.common.pipeline.pipeline_stages_base DEBUG SerializeConnectOpsStage(ConnectOperation): unblocking and releasing queued ops.
2019-10-07 17:57:46,439 azure.iot.device.common.pipeline.pipeline_stages_base INFO  SerializeConnectOpsStage(ConnectOperation): processing 0 items in queue
2019-10-07 17:57:46,439 azure.iot.device.common.pipeline.pipeline_stages_base DEBUG SerializeConnectOpsStage(ConnectOperation): unblock is complete.  completing op that caused unblock
2019-10-07 17:57:46,439 azure.iot.device.common.pipeline.operation_flow ERROR SerializeConnectOpsStage(ConnectOperation): completing with error Unexpected Paho failure during connect
2019-10-07 17:57:46,439 azure.iot.device.common.pipeline.pipeline_thread DEBUG Starting on_complete in callback thread
2019-10-07 17:57:46,439 azure.iot.device.common.async_adapter ERROR Callback completed with error Unexpected Paho failure during connect
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/azure/iot/device/common/mqtt_transport.py", line 252, in connect
    rc = self._mqtt_client.connect(host=self._hostname, port=8883)
  File "/usr/local/lib/python3.6/dist-packages/paho/mqtt/client.py", line 839, in connect
    return self.reconnect()
  File "/usr/local/lib/python3.6/dist-packages/paho/mqtt/client.py", line 962, in reconnect
    sock = socket.create_connection((self._host, self._port), source_address=(self._bind_address, 0))
  File "/usr/lib/python3.6/socket.py", line 704, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  File "/usr/lib/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/azure/iot/device/common/pipeline/pipeline_stages_mqtt.py", line 100, in _execute_op
    self.transport.connect(password=self.sas_token)
  File "/usr/local/lib/python3.6/dist-packages/azure/iot/device/common/mqtt_transport.py", line 255, in connect
    message="Unexpected Paho failure during connect", cause=e
azure.iot.device.common.transport_exceptions.ProtocolClientError: Unexpected Paho failure during connect
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/azure/iot/device/common/mqtt_transport.py", line 252, in connect
    rc = self._mqtt_client.connect(host=self._hostname, port=8883)
  File "/usr/local/lib/python3.6/dist-packages/paho/mqtt/client.py", line 839, in connect
    return self.reconnect()
  File "/usr/local/lib/python3.6/dist-packages/paho/mqtt/client.py", line 962, in reconnect
    sock = socket.create_connection((self._host, self._port), source_address=(self._bind_address, 0))
  File "/usr/lib/python3.6/socket.py", line 704, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  File "/usr/lib/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/azure/iot/device/iothub/aio/async_clients.py", line 29, in handle_result
    return await callback.completion()
  File "/usr/local/lib/python3.6/dist-packages/azure/iot/device/common/async_adapter.py", line 93, in completion
    return await self.future
  File "/usr/local/lib/python3.6/dist-packages/azure/iot/device/common/pipeline/pipeline_stages_mqtt.py", line 100, in _execute_op
    self.transport.connect(password=self.sas_token)
  File "/usr/local/lib/python3.6/dist-packages/azure/iot/device/common/mqtt_transport.py", line 255, in connect
    message="Unexpected Paho failure during connect", cause=e
azure.iot.device.common.transport_exceptions.ProtocolClientError: Unexpected Paho failure during connect

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./main.py", line 57, in <module>
    loop.run_until_complete(main())
  File "/usr/lib/python3.6/asyncio/base_events.py", line 484, in run_until_complete
    return future.result()
  File "./main.py", line 19, in main
    await module_client.connect()
  File "<string>", line 1, in connect
  File "/usr/local/lib/python3.6/dist-packages/azure/iot/device/iothub/aio/async_clients.py", line 97, in connect
    await handle_result(callback)
  File "/usr/local/lib/python3.6/dist-packages/azure/iot/device/iothub/aio/async_clients.py", line 37, in handle_result
    raise exceptions.ClientError(message="Error in the IoTHub client", cause=e)
azure.iot.device.exceptions.ClientError: Error in the IoTHub client

Additionally, I have tried/ensured a few different things:

• Ensured ports 443, 5671, and 8883 are open on edgeHub and bound to the same host ports.
• Ensured corresponding routes exist (although this seems unrelated to my issue).
• Tried using the sync ModuleClient with similar results.
• Tried exposing ports 443, 5671, and 8883 on my Python module.
• Saw that #304 existed, thought perhaps my Ubuntu image lacked system dependencies, and tried installing all dependencies in the provided Dockerfile gist into my Python module.
• Ensured my module is on the same Docker network as edgeHub (network name is azure-iot-edge).

[jackt-moran]

As far as I’m concerned this issue can be closed.

[jackt-moran]

I found the source of the issue and it doesn’t have to do with this library. Sorry about the wild goose chase. It has to do with how I configured my custom Edge Runtime module.

The problem was the “NetworkingConfig” section in my deployment manifest: { "avroProducer": { "version": "0.0.12", "type": "docker", "status": "running", "restartPolicy": "always", "settings": { "image": "${MODULES.avroProducer}", "createOptions": { "HostConfig": {}, "NetworkingConfig": { "EndpointsConfig": { "host": {} } } } } } } Turns out that empty config assigns a module to the host network (who knew). The issue with having a module that communicates with edgeHub on the host network is the domain name “edgeHub” cannot be resolved (IP address would work, but this library uses the domain name instead). When a module is on the default Docker network (“azure-iot-edge”) you can use the domain name “edgeHub” to resolve the edgeHub container without issue.

For example, this JSON manifest for my module put my module back on the default, “azure-iot-edge” network (you may notice I took out “HostConfig” too - I don’t expect that property to have an affect if left in, but I didn’t need it so I took it out): { "avroProducer": { "version": "0.0.1", "type": "docker", "status": "running", "restartPolicy": "always", "settings": { "image": "${MODULES.avroProducer}", "createOptions": {} } } }

So, in conclusion, taking out the “NetworkingConfig” JSON snippet solved my problem. Hopefully this solves your problem too @abouroubi - thank you for the help, @BertKleewein !