'NoneType' object has no attribute 'can_read' at redis.py:670

I run celery on docker, redis on another server. Celery==4.0.0 redis==2.10.5 redis server is v 2.8.4

My celery configuration is very simple, only broker and backend, both are redis. The task I’m running is gunzipping a 10GB file. It takes 10 minutes to finish.

Traceback (most recent call last):
  File "/venv/lib/python3.4/site-packages/celery/worker/worker.py", line 203, in start
    self.blueprint.start(self)
  File "/venv/lib/python3.4/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/venv/lib/python3.4/site-packages/celery/bootsteps.py", line 370, in start
    return self.obj.start()
  File "/venv/lib/python3.4/site-packages/celery/worker/consumer/consumer.py", line 318, in start
    blueprint.start(self)
  File "/venv/lib/python3.4/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/venv/lib/python3.4/site-packages/celery/worker/consumer/consumer.py", line 584, in start
    c.loop(*c.loop_args())
  File "/venv/lib/python3.4/site-packages/celery/worker/loops.py", line 88, in asynloop
    next(loop)
  File "/venv/lib/python3.4/site-packages/kombu/async/hub.py", line 345, in create_loop
    cb(*cbargs)
  File "/venv/lib/python3.4/site-packages/kombu/transport/redis.py", line 1038, in on_readable
    self.cycle.on_readable(fileno)
  File "/venv/lib/python3.4/site-packages/kombu/transport/redis.py", line 337, in on_readable
    chan.handlers[type]()
  File "/venv/lib/python3.4/site-packages/kombu/transport/redis.py", line 670, in _receive
    while c.connection.can_read(timeout=0):
AttributeError: 'NoneType' object has no attribute 'can_read'

[hedleyroos]

The change applied in https://github.com/celery/kombu/commit/66325920c72d395a7f1bafdf97d5462b7ce5b3f8 does not really fix the issue. The problem is the connection property is reset to None the moment a socket is terminated, and the few microseconds between the check if c.connection is not None and the next line while c.connection.can_read(timeout=0) is enough time for this to happen. You will only encounter this on a slow network or a network with packet loss. The two lines need to be threadsafe, and the only quick solution I see is to rewrite it as:

        if c.connection is not None:
        try:
            while c.connection.can_read(timeout=0):
                ret.append(self._receive_one(c))
        except AttributeError:
            pass

This is ugly because AttributeError is such a broad exception. It would be nice to have eg. ConnectionLostError raised in the underlying code.

[alexandernst]

@thedrow Why do you want me to open a new (empty) issue when you have all the details AND the solution (or at least a workaround) to the problem here? Can’t you just re-open this one?