Help: Non deterministic behavior between producers and consumers

Example code:

def test_one(timeout, sleep):
    consumer = KafkaConsumer('test.test_one', group_id=None,
                             bootstrap_servers='localhost:9093')
    producer = KafkaProducer(bootstrap_servers='localhost:9093')

    producer.send('test.test_one', b'test_one')
    time.sleep(sleep)
    p = consumer.poll(timeout)
    print p
    return p == {}

Sometimes the consumer sees the message sent by the producer, sometimes it doesn’t. A non python consumer sees every message sent by the producer, so it is not kafka itself that is the issue, rather it is something in the consumer. sleep was added on the thinking that it could be timing related to poll too soon after the message was sent, but does not seem to have made any impact for a variety of sleep durations, same with adjusting the poll timeout duration.

[dpkp]

producer.send is asynchronous by default; you should do something like this if you want a delivery guarantee producer.send('test.test_one', b'test_one').get(timeout) or just call producer.flush().

Have you also verified that your consumer.poll() is not hitting the timeout? If it is unable to fetch records before the timeout, it will return {}

[jaytailor]

i was also using latest and it caused issues for me as fetcher was in loop forever. When i used earliest it helped me.