producer.flush make celery hang

The following is my test code:

from celery import Celery
from kafka import KafkaProducer

app = Celery('test', broker='redis://127.0.0.1:6379/0')

producer = KafkaProducer(bootstrap_servers=['172.16.24.45:9092', '172.16.24.44:9092'])

@app.task
def send_msg():
    # producer = KafkaProducer(bootstrap_servers=['172.16.24.45:9092', '172.16.24.44:9092'])
    for i in range(10):
        producer.send('test', b'this is the %dth test message' % i)
    producer.flush()


if __name__ == '__main__':
        app.start()

I use the following command to start worker:

celery -A app worker -l debug

then I enter python command line to send task:

from app import *
send_msg.delay()

If I use global producer variable, when I call send_msg.delay(), celery worker will hang and wait for producer flush, it will never end. But If I use local producer variable which is commented in above code, celery worker will work well.

I want to use global producer because It will work more efficient than local and not frequently create and close connections with kafka brokers. But how can I fix this problem?

Please help me and thanks.

[sharkyze]

hey @gzeronet we are having the same issue where the KafkaProducer is not working with multiprocessing, could you please share some resources on how you solved this with gevent ?

[dpkp]

Sorry, that will not work. You cannot share a global producer with celery tasks. You will need to create a local producer for each task instance.

On May 7, 2017 2:20 AM, “starplanet” [e-mail] wrote:

The following is my test code:

from celery import Celeryfrom kafka import KafkaProducer

app = Celery(‘test’, broker=‘redis://127.0.0.1:6379/0’)

producer = KafkaProducer(bootstrap_servers=[‘172.16.24.45:9092’, ‘172.16.24.44:9092’]) @app.taskdef send_msg(): # producer = KafkaProducer(bootstrap_servers=[‘172.16.24.45:9092’, ‘172.16.24.44:9092’]) for i in range(10): producer.send(‘test’, b’this is the %dth test message’ % i) producer.flush()

if name == ‘main’: app.start()

If I use global producer variable, when I call send_msg.delay(), celery worker will hang and wait for producer flush, it will never end. But If I use local producer variable which is commented in above code, celery worker will work well.

I want to use global producer because It will work more efficient than local and not frequently create and close connections with kafka brokers. But how can I fix this problem?

Please help me and thanks.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/dpkp/kafka-python/issues/1098, or mute the thread https://github.com/notifications/unsubscribe-auth/AAzetOBY9SEvNUPEhS26vPd8VbAUyfRbks5r3YzIgaJpZM4NTE0K .