to_kafka throughput

I’m testing to_kafka sink and its throughput is limited by polltime (0.2 sec). Looks like self.producer.poll(0) only polls for one message at a time and so only one callback is called every 0.2 seconds.

This fails:

def test_to_kafka_throughput():
    ARGS = {'bootstrap.servers': 'localhost:9092'}
    with kafka_service() as kafka:
        _, TOPIC = kafka
        source = Stream.from_iterable(range(100)).map(lambda x: str(x).encode())
        kafka = source.to_kafka(TOPIC, ARGS)
        out = kafka.sink_to_list()

        source.start()
        wait_for(
            lambda: len(out) == 100,
            5,
            period=0.1,
            fail_func=lambda: print("len(out) ==", len(out))
        )

The existing test_to_kafka test doesn’t catch this, because it starts waiting on the result only after all the items are emitted.

I spent some time tinkering with the code, but can’t figure out what’s wrong and how to fix this, so any ideas are appreciated.

[roveo]

I think I get it.

The source emits the next item only after the future it awaits on is resolved. But it is resolved only once poll is called and the previous callback is executed, so items are emitted and delivered one by one after each .poll() call, every 0.2 seconds.

[martindurant]

So would you say that it works as intended?

Well, this is certainly an understandable mode, effectively we have a buffer of one. But I don’t think the docstring describes this, so I would say its lacking in at least that. I don’t see any reason that we can’t have messages pile up in the producer buffer, since it will execute the callbacks for us. In the case that we don’t want to reference count, this would be much faster, as @roveo points out, due to batching, but messages might be sent out of order.

There is not much in the documentation about message ordering, except this opaque statement, another gap that needs filling.