"Request Empty body HTTP 599 OpenSSL" error connecting to SQS causes celery worker to hang

Our celery workers have been occasionally logging this openssl connection error with SQS. After logging the error, the worker hangs and won’t accept any more tasks. The traceback seems to indicate this is a problem with kombu and not celery. I’ve tried kombu 4.1.0 and kombu’s master branch and it still happens.

Related celery tickets: https://github.com/celery/celery/issues/3990 https://github.com/celery/celery/issues/3877

Exception: Request Empty body  HTTP 599  OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to us-west-2.queue.amazonaws.com:443  (None)
  File "kombu/async/hub.py", line 136, in fire_timers
    entry()
  File "kombu/async/timer.py", line 68, in __call__
    return self.fun(*self.args, **self.kwargs)
  File "kombu/async/timer.py", line 127, in _reschedules
    return fun(*args, **kwargs)
  File "kombu/async/http/curl.py", line 108, in _timeout_check
    self._process_pending_requests()
  File "kombu/async/http/curl.py", line 132, in _process_pending_requests
    self._process(curl, errno, reason)
  File "kombu/async/http/curl.py", line 178, in _process
    buffer=buffer, effective_url=effective_url, error=error,
  File "vine/promises.py", line 150, in __call__
    svpending(*ca, **ck)
  File "vine/promises.py", line 143, in __call__
    return self.throw()
  File "vine/promises.py", line 140, in __call__
    retval = fun(*final_args, **final_kwargs)
  File "vine/funtools.py", line 100, in _transback
    return callback(ret)
  File "vine/promises.py", line 143, in __call__
    return self.throw()
  File "vine/promises.py", line 140, in __call__
    retval = fun(*final_args, **final_kwargs)
  File "vine/funtools.py", line 98, in _transback
    callback.throw()
  File "vine/funtools.py", line 96, in _transback
    ret = filter_(*args + (ret,), **kwargs)
  File "kombu/async/aws/connection.py", line 233, in _on_list_ready
    raise self._for_status(response, response.read())

software -> celery:4.1.0 (latentcall) kombu:4.1.0 py:3.5.4
            billiard:3.5.0.3 sqs:N/A
platform -> system:Linux arch:64bit imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:sqs results:disabled

TESTING: False
JSONIFY_MIMETYPE: 'application/json'
SESSION_REFRESH_EACH_REQUEST: True
USE_X_SENDFILE: False
LOGGER_NAME: 'lib'
PROPAGATE_EXCEPTIONS: None
BROKER_TRANSPORT_OPTIONS: {
 'queue_name_prefix': 'production-', 'region': 'us-west-2'}
SESSION_COOKIE_PATH: None
SESSION_COOKIE_SECURE: False
PRESERVE_CONTEXT_ON_EXCEPTION: None
PERMANENT_SESSION_LIFETIME: datetime.timedelta(31)
JSON_SORT_KEYS: '********'
TRAP_HTTP_EXCEPTIONS: False
BROKER_URL: 'sqs://********:********@localhost//'
SQLALCHEMY_DATABASE_URI: '********'
SESSION_COOKIE_DOMAIN: None
PREFERRED_URL_SCHEME: 'http'
CELERY_INCLUDE: [*********]
SESSION_COOKIE_NAME: 'session'
DEBUG: False
APPLICATION_ROOT: None
TRAP_BAD_REQUEST_ERRORS: False
EXPLAIN_TEMPLATE_LOADING: False
SEND_FILE_MAX_AGE_DEFAULT: datetime.timedelta(0, 43200)
JSONIFY_PRETTYPRINT_REGULAR: True
DATABASE_URL: '********'
SECRET_KEY: '********'
SQLALCHEMY_TRACK_MODIFICATIONS: False
LOGGER_HANDLER_POLICY: 'always'
SESSION_COOKIE_HTTPONLY: True
TEMPLATES_AUTO_RELOAD: None
MAX_CONTENT_LENGTH: None
JSON_AS_ASCII: True
SERVER_NAME: None
DROPBOX_TOKEN: '********'
TITAN_API_KEY: '********'

[gbataille]

I had the same 599 issue from time on time. (Spoiler, this is unlikely to be a solution to @jmckib problem, but to a similar one)

This was actually due to a bad install (it seems)

To make pycurl compile, we had installed libgnutls28-dev and therefore pycurl used GNUTLS as a TLS backend. It seems according to the wisdom of the web that GNUTLS is less stable than OpenSSL. I removed libgnutls28-dev, pycurl and the pip cache, added libcurl4-openssl-dev and reinstalled pycurl. Now pycurl is using OpenSSL as a back-end.

To check

$ import pycurl
$ pycurl.version

I don’t seem to have 599 issue anymore (for the moment).

I know that the issue of @jmckib talks about OpenSSL and therefore is unlikely to be the same thing, but I thought some people seeing similar 599 errors could have the same issue as me.

[erwan-lemonnier]

As a quick and very dirty fix: I am starting my celery workers in an inifinite while loop, like this:

while true; do celery worker -E -A myapp --concurrency=1 --include myapp.loader; sleep 5; done