KafkaJS and java producer partitioning returning different results

Producing message with the same key gets assigned to different partitions depending if app is using java or nodejs client.

This creates problems when using kafka streams for joining those data - meeting the co-partitioning requirement fails even when message keys are the same.

I created an isolated test code which confirmed it. Simple java and nodejs app with partitioning using murmur2 code extracted from the latest kafkajs and java producer. The code looks the same, but results differ. Even checked that input bytes are equal. ¯_(ツ)_/¯

Test runner result matches the partition number I’m seeing in Kafka broker…

Any idea why and can it be fixed?

Our current workaround is to repartition all kafkajs based messages with java producer…

let key = '100:48069';
let numPartitions = 12;

node bytes {"type":"Buffer","data":[49,48,48,58,52,56,48,54,57]}
node murmur 1713501762
node hash 1713501762
node partition 6

java bytes 49 48 48 58 52 56 48 54 57 
java murmur 1009543857
java hash 1009543857
java partition 9

partitioning test code.zip

[pimpelsang]

Tried it out. Works! Thank you very much for so fast fix

[Nevon]

I couldn’t stop myself, because this is fascinating. I found where the issue is, though I don’t understand it yet.

https://github.com/tulios/kafkajs/blob/c5ab8c229369ae4966ffd3ce22e2b0c00191943b/src/producer/partitioners/default/murmur2.js#L28

I ran some interactive debugging of both the Java and JS implementations, executing them side-by side and comparing the values. On this line, the values diverge between the JS and Java implementations. Given the key you provided, on the first iteration of the loop, k is 976236593 and m is 1540483477 in both JS and Java, but after doing k *= M, the value of k is 2095687045 in Java and 1503876341159274000 in JS.

Essentially:

int k = 976236593
int m = 1540483477

k * m == 2095687045

const k = 976236593
const m = 1540483477

k * m === 1503876341159274000

I’m assuming this is because we’re using the native Number type in Javascript, which has different limits from Java’s int.