Kafka: allow message key and partition to be chosen independently

If a PartitionKeyStrategy is used with a topic, the value is used as the message key, and is then implicitly used to select the partition according to the default behavior of the Kafka client:

If a valid partition number is specified that partition will be used when sending the record. If no partition is specified but a key is present a partition will be chosen using a hash of the key. If neither key nor partition is present a partition will be assigned in a round-robin fashion.

It might be desirable in some cases to control these independently. For example, you might wish to have a message key that is more fine-grained than the partition key, for use with Kafka log compaction on sub-graphs of the entity state.

[datalchemist]

@ignasi35 Yes, my step 1 was indeed derived from @jroper suggestion. But in my step 2, I meant actually using the new property in the lagom kafka-broker internal. Anyway, I think I am on the right line.

I have looked at the details and the implementation seems quite straight forward. But then, I was just a bit puzzled with the inter-dependence between a message key and a partition number. I am not a Kafka expert at all but from what I can see in their Producer API, we can have the following cases:

• no key and no partition number
• one key but no partition number
• one key and one partition number

but we can’t have a partition number without a key.

So, from that, it doesn’t seem good to have two distinct properties like PartitionKeyStrategy and PartitionNumberStrategy because then if the user defines the second without the first we have a problem. A solution I see would be to have a more general strategy encompassing both key and partition generation. e.g.:

trait MessagePartitionStrategy[-Message] {
  def computeMessageKey(message: Message): String
  def computePartitionNumber(message: Message): Option[Int]
}

Hence, it would be possible to define the message key alone, or the key and the partition.

Of course, this would mean more changes because this new type would replace the previous PartitionKeyStrategy (although they could also live together for a while, by giving precedence to the first)

Does such an approach seem suitable ? Do you have another idea to cope with this link between message key and partition ?

[jroper]

The approach we’ve taken to adding properties to a topic mean that this should be straight forward to add without impacting existing APIs and functionality.