Eventlet 0.20.0 breaks dns resolution in nameko

I seem to have hit a very strange bug running nameko inside of docker-compose At this point, i’m able to get around the issue, but I would like to know if eventlet or docker is doing something wrong.

Docker: Version 1.12.3 (13776) Channel: Stable 583d1b8ffe

nameko version 2.3.1

Traceback from inside the container.

[root@ba50a39f5dbb app]# nameko run --config ./cfg/nameko.yaml blah.blah.service:blahService
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/eventlet-0.20.0-py2.7.egg/eventlet/hubs/poll.py", line 115, in wait
    listener.cb(fileno)
  File "/usr/lib/python2.7/site-packages/eventlet-0.20.0-py2.7.egg/eventlet/greenthread.py", line 214, in main
    result = function(*args, **kwargs)
  File "build/bdist.linux-x86_64/egg/nameko/utils.py", line 175, in call
    return getattr(item, name)(*args, **kwargs)
  File "build/bdist.linux-x86_64/egg/nameko/messaging.py", line 191, in setup
    verify_amqp_uri(self.amqp_uri)
  File "build/bdist.linux-x86_64/egg/nameko/amqp.py", line 43, in verify_amqp_uri
    with transport.establish_connection():
  File "/usr/lib/python2.7/site-packages/kombu-4.0.2-py2.7.egg/kombu/transport/pyamqp.py", line 130, in establish_connection
    conn.connect()
  File "/usr/lib/python2.7/site-packages/amqp-2.1.4-py2.7.egg/amqp/connection.py", line 294, in connect
    self.transport.connect()
  File "/usr/lib/python2.7/site-packages/amqp-2.1.4-py2.7.egg/amqp/transport.py", line 120, in connect
    self._connect(self.host, self.port, self.connect_timeout)
  File "/usr/lib/python2.7/site-packages/amqp-2.1.4-py2.7.egg/amqp/transport.py", line 150, in _connect
    host, port, 0, socket.SOCK_STREAM, SOL_TCP,
  File "/usr/lib/python2.7/site-packages/eventlet-0.20.0-py2.7.egg/eventlet/support/greendns.py", line 464, in getaddrinfo
    raise socket.gaierror(socket.EAI_NONAME, 'No address found')
gaierror: [Errno -2] No address found

The host dns resolves with a non authoritive response, but all dns functions work correctly from the container itself.

Pinning the version of eventlet to 0.16.1 seems to stop the issue from occurring.

Is the eventlet dns client implementation known to have issues?

thanks in advance! 😃 Rob

[mattbennett]

There are a couple of threads of conversation happening here now. Let me try to summarise:

1. Kombu 4.0 requires eventlet 0.2.0, but nameko pins kombu to 3.0.37, so this is irrelevant. #378 is not related.

2. Eventlet 0.2.0 bundles dnspython and seems to use it to monkeypatch the socket library’s DNS functions. Unfortunately dnspython doesn’t have exactly the same behaviour – in particular, it won’t resolve a fully qualified domain name without a . at the end if there’s a search domain specified. From what I understand about DNS that’s the correct behaviour, but the fact that socket.getaddrinfo behaves differently suggests there’s bug is in eventlet’s assumption that they’re equivalent.

[kooba]

Hi @manfred-chebli & @rgardam have you docker “linked” your broker container to your service container? Or do you just rely on Docker embedded DNS server by simply using the name (broker in your case) of the container?

Once we’ve upgraded to master version of Eventlet (before they cut 0.20 release, we needed it to use a fix to HTTPS body parsing https://github.com/eventlet/eventlet/blob/master/NEWS#L8) we noticed that Docker embedded DNS name resolution does not work any more.

Since eventlet vendored dnspython I’ve opened issue directly with dnspython here https://github.com/rthalley/dnspython/issues/219 but they said this is expected behaviour. There is a gist to reproduce the issue.

The workaround we’ve employed for now is to trick dnspython into thinking that our container names are fully qualified domains by adding period . to the url. In your case it would be amqp://user:pass@broker.:5672

I think by using dnspython for dns resolution eventlet broke some functionality.