FileNotFoundError in publish

I’m running into this problem using dramatiq. As I have been digging into it, I feel like it’s more likely a pika issue. Here is the stacktrace:

FileNotFoundError: [Errno 2] No such file or directory
  File "flask/app.py", line 2309, in __call__
    return self.wsgi_app(environ, start_response)
  File "appoptics_apm/middleware.py", line 142, in __call__
    result = self.wrapped_app(environ, wrapped_start_response)
  File "werkzeug/contrib/fixers.py", line 152, in __call__
    return self.app(environ, start_response)
  File "raven/middleware.py", line 100, in __call__
    iterable = self.application(environ, start_response)
  File "flask/app.py", line 2295, in wsgi_app
    response = self.handle_exception(e)
  File "flask_restful/__init__.py", line 273, in error_router
    return original_handler(e)
  File "flask/app.py", line 1741, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "flask/_compat.py", line 35, in reraise
    raise value
  File "flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "flask_restful/__init__.py", line 273, in error_router
    return original_handler(e)
  File "flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "flask/_compat.py", line 35, in reraise
    raise value
  File "flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "flask/views.py", line 88, in view
    return self.dispatch_request(*args, **kwargs)
  File "flask/views.py", line 158, in dispatch_request
    return meth(*args, **kwargs)
  File "flask_login/utils.py", line 261, in decorated_view
    return func(*args, **kwargs)
  File "sylo/permissions/access.py", line 96, in wrapped
    return f(*args, **kwargs)
  File "webargs/core.py", line 447, in wrapper
    return func(*new_args, **kwargs)
  File "<my-file>", line 47, in wrapper
    result = func(*args, **kwargs)
  File "<my-file>", line 486, in func
    <my_actor>.send(one_id, two_id)
  File "dramatiq/actor.py", line 179, in send
    return self.send_with_options(args=args, kwargs=kwargs)
  File "dramatiq/actor.py", line 198, in send_with_options
    return self.broker.enqueue(message, delay=delay)
  File "dramatiq/brokers/rabbitmq.py", line 250, in enqueue
    properties=properties,
  File "pika/adapters/blocking_connection.py", line 2206, in publish
    immediate=immediate)
  File "pika/channel.py", line 425, in basic_publish
    (properties, body))
  File "pika/channel.py", line 1328, in _send_method
    self.connection._send_method(self.channel_number, method, content)
  File "pika/connection.py", line 2341, in _send_method
    self._send_message(channel_number, method, content)
  File "pika/connection.py", line 2355, in _send_message
    self._send_frame(frame.Method(channel_number, method_frame))
  File "pika/connection.py", line 2327, in _send_frame
    self._flush_outbound()
  File "pika/adapters/base_connection.py", line 330, in _flush_outbound
    self._manage_event_state()
  File "pika/adapters/base_connection.py", line 536, in _manage_event_state
    self.event_state)
  File "pika/adapters/select_connection.py", line 446, in update_handler
    self._poller.update_handler(fileno, events)
  File "pika/adapters/select_connection.py", line 639, in update_handler
    events_to_set=events_set)
  File "pika/adapters/select_connection.py", line 1152, in _modify_fd_events
    self._poll.modify(fileno, events)

It is happening pretty regularly when running flask in uwsgi with threads enabled. Any ideas?

_Originally posted by @Nizebulous in https://github.com/pika/pika/issues/1105#issuecomment-461065451_

[booleys1012]

I am including this note just to supplement the issue with additional information in case it is helpful.

python2.7 pika==0.13.0 RabbitMQ 3.6.10, Erlang R16B03-1

I have a web application (django via uwsgi) and multiple python 2.7 microservices publishing to RabbitMQ using the same code (using pika). I am not using any python threading related to publishing or subscribing – this includes not using threading in uwsgi. This has been operating for approximately 2 years without seeing the error logs mentioned above.

Docker migration I recently migrated all services to container deployments (docker) as a proof-of-concept and I see this error now after a handful of minutes of successful publishing, but only on the webserver – the microservices do not appear to exhibit this error

No code changes, just the migration to docker

Update (2019-02-27) I am starting to think it was related to uwsgi and the --lazy-apps flag, which I did turn off as part of the above docker release (apologies). I am thinking the difference is creating a pika connection pre-fork in uwsgi versus post-fork. Creating the connection pre-fork gives any forked processes an (illegal) copy of the connection (file descriptor), which could explain the IOError. This would negate the implications that this is related to docker. I’ll add further updates if I find anything more.

[Nizebulous]

And just for further information…I was able to replicate this running the same docker image on my local mac.