KafkaProducer hangs when publishing to new topic (and using uWSGI master process)

I’m trying to upgrade from SimpleProducer to the newer KafkaProducer, but I’m running into an issue where the producer hangs when it tries to publish to a topic that doesn’t exist. This occurs only when the producer is run with uWSGI, and uWSGI is configured to use a master process.

The messages are successfully published if uWSGI is run without a master process or if the topic already exists.

This behavior manifests with both kafka 0.9.0.1 and 0.10.0.

It seems like this might be related to http://stackoverflow.com/questions/37693563/how-to-make-kafka-python-or-pykafka-work-as-an-async-producer-with-uwsgi-and-gev and https://github.com/dpkp/kafka-python/issues/709, but the fact that it only breaks for nonexistent topics makes me think it might be a separate issue.

[dpkp]

Ah, thank you! Two issues:

You cannot share producer instances across processes, only threads. I expect that is why the master process pattern is failing.

Second, producer.send() is async but is not guaranteed to deliver if you close the producer abruptly. In your final example I suspect that your producer instances are so short-lived that they are being reaped before flushing all pending messages. To guarantee delivery (or exception) call producer.send().get(timeout) or producer.flush() . otherwise you’ll need to figure out how to get a producer instance per-uwsgi-thread and have it shared across requests (you would still want to flush before thread shutdown to guarantee no messages are dropped)

[zhangyufei49]

uWsgi with ‘master=false’ option works fine for this scene. But I solved this by create independent producer contexts in each process. I write a ‘mq’ module

# __init__.py

from mq.MsgQueue import MsgQueue
from flask import current_app


def _init(app):
    uri = app.config['MQ_URI']
    return MsgQueue(uri)


mq_context = _init(current_app)

# MsgQueue.py

from kafka import KafkaProducer
from ujson import dumps


class MsgQueue(object):
    def __init__(self, uri):
        super(MsgQueue, self).__init__()
        self.producer = KafkaProducer(
            bootstrap_servers=uri, api_version=(0, 10, 2))

    def send(self, topic, data):
        self.producer.send(topic, dumps(data, ensure_ascii=False).encode())

And in app logic:

from mq import mq_context  # ensure only  load 'mq_context' once in each process
mq_context.send(topic, data)

pykafka also needs flow this way