Producer compression_type ignored for api_version >= 0.11

Is there a point in ignoring producer compression_type for api version >= 0.11?

I am trying to use a producer with gzip compression like so

producer = kafka.KafkaProducer(bootstrap_servers=settings.KAFKA_BOOSTRAP_SERVERS, compression_type='gzip')

future = producer.send(
    'test',
    value=json.dumps(data).encode('utf-8'),
    key=b'test'
)

future.get(timeout=60)

When the message is larger than 1MB i get kafka.errors.MessageSizeTooLargeError.

Digging deeper, kafka.producer.kafka.KafkaProducer has a method for estimating message size.

    def _max_usable_produce_magic(self):
        if self.config['api_version'] >= (0, 11):
            return 2
        elif self.config['api_version'] >= (0, 10):
            return 1
        else:
            return 0

    def _estimate_size_in_bytes(self, key, value, headers=[]):
        magic = self._max_usable_produce_magic()
        if magic == 2:
            return DefaultRecordBatchBuilder.estimate_size_in_bytes(
                key, value, headers)
        else:
            return LegacyRecordBatchBuilder.estimate_size_in_bytes(
                magic, self.config['compression_type'], key, value)

We have an api version 1.0.0, so compression_type is ignored when estimating message size.

I am using kafka-python==1.4.4.

Can you please explain a bit what is going on?

[tvoinarovskyi]

Kafka-python in regards to message size estimation works only on non-compressed messages (basically that is always exact size of the message). Compressed message’s size is hard to predict and Java does a lot of strange things to allow more efficient batches (like trying to send a too big batch and reformat it and resend in case of failure). Therefore I left the logic that only operates on uncompressed batches. If you need to send 1 MB of data you need to set limits as if it’s not compressed. Sorry for the inconvienience, but doing compression estimation and resplitting batches on failure does not seem like an easy thing to do, at least I don’t see much benefit in it for now.

[alexeysofin]

Thank you for clarifying!