Unrecoverable error: ValueError('I/O operation on closed epoll fd',)

Kombu 3.0.8

Unrecoverable error: ValueError('I/O operation on closed epoll fd',)

Stacktrace (most recent call last):

  File "celery/worker/__init__.py", line 212, in start
    self.blueprint.start(self)
  File "celery/bootsteps.py", line 123, in start
    step.start(parent)
  File "celery/bootsteps.py", line 373, in start
    return self.obj.start()
  File "celery/worker/consumer.py", line 270, in start
    blueprint.start(self)
  File "celery/bootsteps.py", line 123, in start
    step.start(parent)
  File "celery/worker/consumer.py", line 596, in start
    replies = I.hello(c.hostname, revoked._data) or {}
  File "celery/app/control.py", line 111, in hello
    return self._request('hello', from_node=from_node, revoked=revoked)
  File "celery/app/control.py", line 70, in _request
    timeout=self.timeout, reply=True,
  File "celery/app/control.py", line 307, in broadcast
    limit, callback, channel=channel,
  File "kombu/pidbox.py", line 285, in _broadcast
    channel=chan)
  File "kombu/pidbox.py", line 324, in _collect
    self.connection.drain_events(timeout=timeout)
  File "kombu/connection.py", line 279, in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
  File "kombu/transport/virtual/__init__.py", line 830, in drain_events
    item, channel = get(timeout=timeout)
  File "kombu/transport/redis.py", line 292, in get
    self._register_BRPOP(channel)
  File "kombu/transport/redis.py", line 240, in _register_BRPOP
    self._register(*ident)
  File "kombu/transport/redis.py", line 229, in _register
    self.poller.register(sock, self.eventflags)
  File "kombu/utils/eventio.py", line 78, in register
    self._epoll.register(fd, events)

[andrewwatts]

While using celery our temporary solution is to use the gevent pool (–pool=gevent), however that solution is listed as experimental, so not sure we’ll ship it to production, but it works while debugging other issues within our celery tasks.

I encountered this error while testing what happens to celery when failing nodes in our rabbitmq cluster. It appears, the file descriptor is being re-used after closing the first connection. This appears to only happen with epoll and also was not able to reproduce on OS X using select.

[karanratra]

with reference to [https://github.com/celery/kombu/issues/706] @andrewwatts @ask @draskomikic I am also encountering the same problem when I started testing the rabbitmq cluster and shutdown the master broker node. Worker started giving the same error (shown below) and I tried the temp solution that you provided --> the gevent pool (–pool=gevent).

It fixed the problem but then I noticed that the concurency option is not working (–concurrency=4) I need to use the concurrency method for my worker and donot want to go with gevent threads.

Can you please tell me what is the workaround so that the cluster fail over works as well as I can also use concurrency option.

Here is the error I am getting

_2018-08-01 16:37:41,559: ERROR/MainProcess] Unrecoverable error: ValueError(‘I/O operation on closed epoll fd’,) Traceback (most recent call last): File “/usr/lib/python2.6/site-packages/celery-3.1.5-py2.6.egg/celery/worker/init.py”, line 212, in start self.blueprint.start(self) File “/usr/lib/python2.6/site-packages/celery-3.1.5-py2.6.egg/celery/bootsteps.py”, line 123, in start step.start(parent) File “/usr/lib/python2.6/site-packages/celery-3.1.5-py2.6.egg/celery/bootsteps.py”, line 373, in start return self.obj.start() File “/usr/lib/python2.6/site-packages/celery-3.1.5-py2.6.egg/celery/worker/consumer.py”, line 270, in start blueprint.start(self) File “/usr/lib/python2.6/site-packages/celery-3.1.5-py2.6.egg/celery/bootsteps.py”, line 123, in start step.start(parent) File “/usr/lib/python2.6/site-packages/celery-3.1.5-py2.6.egg/celery/worker/consumer.py”, line 467, in start c.connection = c.connect() File “/usr/lib/python2.6/site-packages/celery-3.1.5-py2.6.egg/celery/worker/consumer.py”, line 369, in connect conn.transport.register_with_event_loop(conn.connection, self.hub) File “/usr/lib/python2.6/site-packages/kombu-3.0.6-py2.6.egg/kombu/transport/pyamqp.py”, line 124, in register_with_event_loop loop.add_reader(connection.sock, self.on_readable, connection, loop) File “/usr/lib/python2.6/site-packages/kombu-3.0.6-py2.6.egg/kombu/async/hub.py”, line 214, in add_reader return self.add(fds, callback, READ | ERR, args) File “/usr/lib/python2.6/site-packages/kombu-3.0.6-py2.6.egg/kombu/async/hub.py”, line 165, in add self.poller.register(fd, flags) File “/usr/lib/python2.6/site-packages/kombu-3.0.6-py2.6.egg/kombu/utils/eventio.py”, line 78, in register self.epoll.register(fd, events) ValueError: I/O operation on closed epoll fd

I am using celery3.1.5